Closed yarikoptic closed 1 year ago
@yarikoptic and @jwodder - did you use this kind of session creation and awaiting like this example? https://filesystem-spec.readthedocs.io/en/latest/async.html#using-from-async
@satra No. We use datalad-fuse, which does not have any direct async functionality.
and we are "doomed" to go through datalad-fuse
since we want to test matlab libraries... and who knows what else later which also might not be easily interfaceable via fsspec directly.
fsspec has issues with parallel/multi-threaded execution. As a workaround we can for now just make it all serial within each dandiset if there is an easy way for
async
to pretty much becomesync
, i.e. not to proceed to the next asset until it is done with the previous one.