Closed yarikoptic closed 1 year ago
@yarikoptic How many Dandisets should be processed at once?
@yarikoptic Ping.
should be a CLI option. I don't know what would be the optimal -- may be we could scale to number of CPUs or above! Default -- I don't care, could be 1
(no parallelization)
As locking at fsspec/datalad-fuse level virtually causes serial processing within each dandiset, I think we should parallelize across dandisets, so that with locking per each dandiset at fsspec level we would still be parallel across them and have reasonablish run time overall.