Can't retrive by md.read

[eromoe]

After debugging, I found ts.get can retrive records, but ts.mrange return empty list .

self.ts.get('markets:cn:000001.sz:raw:amount')
> (1577871000000, 0.08393649481810428)
params
> {'from_time': '-', 'to_time': '+', 'with_labels': True, 'filters': ['tl=markets', 'c1=cn', 'c2=000001.sz', 'timeframe=raw'], 'count': 1000}
self.ts.mrange(**params)
> []

I think the problem may come from the creation of filters , I don't know.

[eromoe]

OK, seems there is a lot of case problems in this package ... not safe for upcase .

[ahmadazizi]

provide the full code to reproduce the issue, it look likes the problem is from your code, not the packge. why you've used dots in your classifier names?

[eromoe]

Because the raw symbol name is like that, upcase with dots . Upcase resulted in that problem, just check c2 in the image I posted is still upcased .

Have to force symbol name to lower.

import time, datetime, random
from pytz import timezone

from redis_timeseries_manager import RedisTimeseriesManager

settings = {
    'host': 'localhost',
    'port': 6379,
    'db': 13,
    'password': None,
}

class MarketData(RedisTimeseriesManager):
    _name = 'markets'
    _lines = ['open', 'high', 'low', 'close', 'volume', 'amount']
    _timeframes = {
        'raw': {'retention_secs': 60*60*24*3}, # retention 3 days
        '1m': {'retention_secs': 60*60*24*7, 'bucket_size_secs': 60}, # retention 7 day; timeframe 60 secs
    }

    #compaction rules
    def _create_rule(self, c1:str, c2:str, line:str, timeframe_name:str, timeframe_specs:str, source_key:str, dest_key:str):
        if line == 'open':
            aggregation_type = 'first'
        elif line == 'close':
            aggregation_type = 'last'
        elif line == 'high':
            aggregation_type = 'max'
        elif line == 'low':
            aggregation_type = 'min'
        elif line == 'volume':
            aggregation_type = 'sum'
        elif line == 'amount':
            aggregation_type = 'sum'
        else:
            return
        bucket_size_secs = timeframe_specs['bucket_size_secs']
        self._set_rule(source_key, dest_key, aggregation_type, bucket_size_secs)

from collections import OrderedDict

def generate_tick(ts):
    keys = ['time', 'open','high','low','lastPrice','volume','amount']
    vals = [ts] + list(np.random.random(4)*20) + list(np.random.random(2)*100)
    return dict(zip(keys, vals))

data = OrderedDict()
# sample_codes = codes.sample(np.random.randint(500, 700))
sample_codes = ['000001.SZ',
 '000002.SZ',
 '000004.SZ',
 '000006.SZ',
 '000007.SZ',
 '000008.SZ',
 '000009.SZ',
 '000010.SZ',
 '000011.SZ',
 '000012.SZ']

# mock data
for ts in range(unix_timestamp('2020-01-01 09:30:00', unit='ms'), unix_timestamp('2020-01-01 11:30:00', unit='ms'), 1000):
    data[ts] = {c: generate_tick(ts) for c in sample_codes}

# insert
for ts, rows in data.items():
    r = md.insert(
        # time, code,  o, h , l , c, v, a
        data=[[v['time'], k.lower(), v['open'], v['high'], v['low'], v['lastPrice'], v['volume'], v['amount']] for k, v in rows.items()],
        c1='cn',
        c2_position=1,
        create_inplace=True,
    )
    if not r[0]: print(r[1])

d = md.read(
    c1='cn',
    c2=sample_codes.iloc[0].lower(),
    timeframe='raw',
)
d[1][:2]

Above code works. But another problem: read 1m frame still return nothing .

d = md.read(
    c1='cn',
    c2=sample_codes.iloc[0].lower(),
    timeframe='1m',
)
d

PS: I removed some * 1000 in code, since my data is ms based.

[ahmadazizi]

1. Because the combination of uppercase and lowercase is troublesome, the package tries to convert all inputs to lowercase , but it may not apply in some places. So, use the lowercase letters everywhere while naming.
2. options for time precision is planned.
3. your issue is because your're using db 13 for redis. As mentioned in docs, redis has some bugs that compaction rules does not work on dbs other than 0