dandi / zarr_checksum

Algorithms for calculating a zarr checksum against local or cloud storage
https://pypi.org/project/zarr-checksum/
Apache License 2.0
11 stars 3 forks source link

Make traversal of the zarr + checksumming interleaved via threading #52

Open yarikoptic opened 10 months ago

yarikoptic commented 10 months ago

traversal of folder could be taking awhile since filesystem itself could be slow. It was initially advocated by @satra and later implemented in dandi-cli to get checksumming of zarr folders to become more efficient through doing traversal and checksumming in separate threads, as e.g. done in https://github.com/dandi/dandi-cli/blob/72816ab51a4f168bd9629cc6ac9a6a4edd585bee/dandi/support/digests.py#L101