search

man-group / arctic

High performance datastore for time series and tick data
https://arctic.readthedocs.io/en/latest/
GNU Lesser General Public License v2.1
3.06k stars 583 forks source link

Have a restore_version api that supports operations with uncompressed chunks #830

Open shashank88 opened 4 years ago

shashank88 commented 4 years ago

Currently in restore_version we read and write the entire data to a new version. There was a more efficient implementation of this but that was reverted as it might cause corruptions.

What we want is to have a version that just reads and writes uncompressed chunks to save up on memory blowing up due to the new bytearray + collecting uncompressed data chunks which we ran into.

shashank88 commented 4 years ago

https://github.com/man-group/arctic/issues/579 was the issue caused last time