I think it should entail just using few more config variables to cache for each zarr (probably could be applied to each dandiset as well but no need ATM) following config variables with stats from prior run
dandi.stats -- comma separated (str, int, int) - (commit hexsha, number of files, size in bytes)
if defined, loaded, commit compared to current one , and if not different - stored stats are used. If different hexsha, or anything goes wrong with parsing the tuple - recompute and store new value.
the need was identified in https://github.com/dandi/dandisets/pull/268#issuecomment-1259889571
I think it should entail just using few more config variables to cache for each zarr (probably could be applied to each dandiset as well but no need ATM) following config variables with stats from prior run
dandi.stats-- comma separated (str, int, int) - (commit hexsha, number of files, size in bytes)if defined, loaded, commit compared to current one , and if not different - stored stats are used. If different hexsha, or anything goes wrong with parsing the tuple - recompute and store new value.
So should replace the proposed condition in https://github.com/dandi/dandisets/pull/268/files#diff-166e4f15f34fb7f16938bb2797f2766a47f58ac27b24099a9f6261956a4485e2R475