search

man-group / arctic

High performance datastore for time series and tick data
https://arctic.readthedocs.io/en/latest/
GNU Lesser General Public License v2.1
3.04k stars 583 forks source link

Use lz4.frame #627

Open bmoscon opened 5 years ago

bmoscon commented 5 years ago

Its not that hard to exceed lz4 memory constraints or or cause it to overflow. The solution is to use lz4.frame which lets you compress a block of data in chunks.

fl4p-old commented 5 years ago

Isn't block size capped to 16MB anyway due to mongodb constraint?

bmoscon commented 5 years ago

true, but data is serialized, compressed and then broken up into 16MB chunks for storage in mongo, so the size constraint is currently on Lz4's block size, which is why we should move to frame.

bartmostert1 commented 5 years ago

Every time i want to use arctic i get the following errror

No module named 'lz4._version'

already removed arctic and re install it throw pip but still it's not working

bmoscon commented 5 years ago

@bartmostert1 that seems unrelated to this feature request. Please open a new ticket and specify the operating system and give the ouput of pip freeze