aiidateam / disk-objectstore

An implementation of an efficient "object store" (actually, a key-value store) writing files on disk and not requiring a running server
https://disk-objectstore.readthedocs.io
MIT License
15 stars 8 forks source link

Proof-of-concept implemenation for supporting multiple pack repositories #126

Closed zhubonan closed 2 years ago

zhubonan commented 2 years ago

This PR allows the packs to be stored in additional locations (pack repositories) other than the main object store folder.

The main reason for allowing this is allow packs to be moved to different file systems (slow but spacious ones) to free up spaces in the main filesystem (usually on a SSD), in case of growing demand of storage space.

Packing loose objects and writing directly to the packs will only take place on the main storage to avoid any performance impact.

One exception is when repacking - this will be done directly inside the pack repository (on the same filesytem as the pack file to be repacked). This means there is no longer a single path for temporaray pack that is used for repacking (previously pack_id=-1), as its path on the file system now depends on the id of the pack that is being repacked.

The user is resposible for moving the packs, ideally when the repository is fully "offline". Online operation is still possible, but requires a sequence of copy-rename-unlink operations.

Todos:

codecov[bot] commented 2 years ago

Codecov Report

Merging #126 (721df95) into develop (7a09ea2) will decrease coverage by 0.73%. The diff coverage is 84.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #126      +/-   ##
===========================================
- Coverage    99.52%   98.79%   -0.74%     
===========================================
  Files            8        8              
  Lines         1675     1736      +61     
===========================================
+ Hits          1667     1715      +48     
- Misses           8       21      +13     
Impacted Files Coverage Δ
disk_objectstore/container.py 97.98% <84.00%> (-1.42%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 7a09ea2...721df95. Read the comment docs.

zhubonan commented 2 years ago

close since it is superseded by #133