Open yarikoptic opened 2 years ago
@yarikoptic The threaded walk is not used during uploads; it's only used to verify downloads and by dandi digest
. Also, the results in https://github.com/con/fscacher/pull/67 indicate that the threaded walk is not an improvement when you're just listing files; its value in Zarr digesting comes from being able to calculate MD5 hashes of files concurrenctly.
hm, interesting... may be I was under wrong impression that it does provide some benefits, will need to recall details better.
We have https://github.com/dandi/dandi-cli/blob/master/dandi/support/threaded_walk.py which is used during zarr uploads. As now we do have Datalad datasets for (some/most) of zarr folders, we better add exclusion of
.git
,.datalad
, .dotdirs etc folders there as we do infind_files
. Ideally RF should then include RFing offind_files
to use threaded_walk with those added exclusion features so we gain speed up in places where we usefind_files