datalad-fuse via fsspec caching is too slow - needs a solution

dandi / dandisets-healthstatus

Healthchecks of dandisets and support libraries (pynwb and matnwb)

0 stars 1 forks source link

datalad-fuse via fsspec caching is too slow - needs a solution #53

Closed yarikoptic closed 3 days ago

yarikoptic commented 1 year ago

that has to be addressed one way (contributing to fsspec) or another (reimplementing fsspec within datalad-fuse while resolving slow fsspec caching issue) to get some sensible performance. ATM it is slow since sparse caching seems to write upon every block read/write into index, and index is also shared across all files, so it all is VERY slow on any sizeable dataset.

bendichter commented 1 year ago

do we need caching?

yarikoptic commented 1 year ago

i would have assumed that it should make things faster since we would be accessing the same portions of the files over and over again (different tests/versions), so caching should be good.

yarikoptic commented 5 months ago

The most likely solution is the freshly developed dandidav and then some webdav FUSE with or without any caching. Related:

issue: #66
implementation: #67

Another possible could potentially be to turn off caching in datalad-fuse:

https://github.com/datalad/datalad-fuse/issues/103 although I am not 100% certain that it would be a thread-safe solution.

jwodder commented 3 days ago

As of #77, we no longer use datalad-fuse.