dandi / dandisets-healthstatus

Healthchecks of dandisets and support libraries (pynwb and matnwb)
0 stars 1 forks source link

Serialize processing within each dandiset #24

Closed yarikoptic closed 1 year ago

yarikoptic commented 1 year ago

fsspec has issues with parallel/multi-threaded execution. As a workaround we can for now just make it all serial within each dandiset if there is an easy way for async to pretty much become sync , i.e. not to proceed to the next asset until it is done with the previous one.

satra commented 1 year ago

@yarikoptic and @jwodder - did you use this kind of session creation and awaiting like this example? https://filesystem-spec.readthedocs.io/en/latest/async.html#using-from-async

jwodder commented 1 year ago

@satra No. We use datalad-fuse, which does not have any direct async functionality.

yarikoptic commented 1 year ago

and we are "doomed" to go through datalad-fuse since we want to test matlab libraries... and who knows what else later which also might not be easily interfaceable via fsspec directly.