Suggested by @giovannipizzi:
Currently, verdi storage maintain in aiida-core might take a long time to proceed, and many users might not be patient enough, and they may cancel the progress just very close to the end. It's nice to provide an estimated time so they will have an idea of how long this would take.
To implement it, I followed these steps:
1) verdi storage maintain calls here in the disk object store backend
2) which would possibly call either of these three methods: a) pack_all_loose b) repack c) clean_storage
3) I made a small container (300MB, ~600 files) and ran cProfile to look for the time-consuming parts and inserted "update" in the relevant positions:
pack_all_loose: the iterations over loose_objects
repack: mostly the iteration, but also safe_flush_to_disk which calls os.fsync, there is no easy way to predict time consumed on that, so I just ignore it, which means the maintenance will still go on a bit even after the bar reaches 100%.
clean_storage: was very quick for the mentioned size of container—just a few milliseconds—so I trusted the scalability and skipped adding a bar for this method.
Once this is approved, I can make another PR on aiida-core to pass a progress bar to these.
Suggested by @giovannipizzi: Currently,
verdi storage maintain
inaiida-core
might take a long time to proceed, and many users might not be patient enough, and they may cancel the progress just very close to the end. It's nice to provide an estimated time so they will have an idea of how long this would take. To implement it, I followed these steps: 1)verdi storage maintain
calls here in the disk object store backend 2) which would possibly call either of these three methods: a)pack_all_loose
b)repack
c)clean_storage
3) I made a small container (300MB, ~600 files) and rancProfile
to look for the time-consuming parts and inserted "update" in the relevant positions:pack_all_loose
: the iterations overloose_objects
repack
: mostly the iteration, but alsosafe_flush_to_disk
which callsos.fsync
, there is no easy way to predict time consumed on that, so I just ignore it, which means the maintenance will still go on a bit even after the bar reaches 100%.clean_storage
: was very quick for the mentioned size of container—just a few milliseconds—so I trusted the scalability and skipped adding a bar for this method.Once this is approved, I can make another PR on
aiida-core
to pass a progress bar to these.