search

man-group / arctic

High performance datastore for time series and tick data
https://arctic.readthedocs.io/en/latest/
GNU Lesser General Public License v2.1
3.05k stars 583 forks source link

Garbage collection for VersionStore #869

Closed qiuwei closed 4 years ago

qiuwei commented 4 years ago

Arctic Version

1.79.3

Arctic Store

VesionStore

Platform and version

Ubuntu 18.04

Description of problem and/or code sample that reproduces the issue

I was writing a huge DataFrame(about 40Gigabytes) into a version store. For some reason, the writing process crashed. The symbol does not show up in library.list_symbols(). However, disk space is still used. How can I clean up the partially written data?

shashank88 commented 4 years ago

I think fsck should be able to catch this if not there are 2 things you can do: a) If the lib had just one symbol you can delete the library which would, in turn, delete the collections which have the leaked data chunks b) Go to mongo and run a delete query for that symbol as regex

qiuwei commented 4 years ago

Thanks for the tips!