aiidateam / disk-objectstore

An implementation of an efficient "object store" (actually, a key-value store) writing files on disk and not requiring a running server
https://disk-objectstore.readthedocs.io
MIT License
15 stars 8 forks source link

Progress bar for `repack` and `pack_all_loose` #171

Open khsrali opened 2 months ago

khsrali commented 2 months ago

Suggested by @giovannipizzi: Currently, verdi storage maintain in aiida-core might take a long time to proceed, and many users might not be patient enough, and they may cancel the progress just very close to the end. It's nice to provide an estimated time so they will have an idea of how long this would take. To implement it, I followed these steps: 1) verdi storage maintain calls here in the disk object store backend 2) which would possibly call either of these three methods: a) pack_all_loose b) repack c) clean_storage 3) I made a small container (300MB, ~600 files) and ran cProfile to look for the time-consuming parts and inserted "update" in the relevant positions:

Once this is approved, I can make another PR on aiida-core to pass a progress bar to these.

khsrali commented 3 days ago

Note: test fails because setup-python@v1 is deprecated. A solution is provided in PR #172