search

aiidateam / disk-objectstore

An implementation of an efficient "object store" (actually, a key-value store) writing files on disk and not requiring a running server
https://disk-objectstore.readthedocs.io
MIT License
15 stars 8 forks source link

Improve performance of list_all_objects for packed objects #71

Closed giovannipizzi closed 4 years ago

giovannipizzi commented 4 years ago

For millions of packed objects, the call was getting 'stuck' trying to first fetch all data from the DB. Now we query the DB in chunks of 1000 elements (we could improve the chunk size later, I added a performance test so we can monitor this)

Fixes #69

codecov[bot] commented 4 years ago

Codecov Report

Merging #71 into develop will not change coverage. The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##           develop       #71   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            7         7           
  Lines          976       982    +6     
=========================================
+ Hits           976       982    +6
Impacted Files Coverage Δ
disk_objectstore/container.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 801df0c...03ba56b. Read the comment docs.