Closed gkorland closed 4 years ago
@gkorland Can we discuss the pros and cons on this? @danni-m can you chime in?
lets look at the current serialization: Cons:
with compressed RDB: Cons:
I think that especially if we want to support RoF we must make sure that load/store is fast and that the data on disk is not 20 times bigger than in memory
As for backward, I think it's manageable with the right tests and RDB versions
I have conducted a simplistic test of saving and reloading about 200MB in both RedisBloom which loads raw data and RedisTimeSeries which currently processes the data. Bloom was about 28 times faster (0.3 vs 8.4 seconds). While TimeSeries will take longers as it pushes the chunks into the dictionary so we can't expect to reach that speed, It will be within the same range.
Checked a few more settings - The previous results (8.4 s) were for compressed floats. Uncompressed floats took 2.8 s. (x9) Compressed integers took about 30 s or 100 times longer than bloom.
I'm using RedisTimeSeries to store IOT data (unix timestamp, integer value). My RDB file (uncompressed) is about 12 times my memory size (compressed). It would be wonderful if we can store/load compressed data straight to/from RDB (or maybe another file). Or having a mode setting in redis.conf where we can do this without backward compatibility. I can run an instance of redis just for timeseries data that don't need backward compatibility.
Store data to disk in in compressed mode