ATM local filesystem caching of accessed content is non-optional. To make fsspec work multi-threaded, heavy locking was introduced and we are using custom branch with that. That locking renders access very slow. As in many use cases network access could be almost as fast as the drive access or even faster (e.g. on an EC2 instance), it might make sense to make such caching optional, or even not enabled by default at all so we could use non-patched fsspec with default installation of datalad-fuse.
So let's add --caching none,ondisk with none as default, and provide documentation in e.g. README to say that patched version of fsspec is needed for ondisk.
NB while coming up with interface, might be worth considering that later we might make caching support configurable as well, so "ondisk" might not be descriptive enough. But we could think about it whenever we get there.
ATM local filesystem caching of accessed content is non-optional. To make fsspec work multi-threaded, heavy locking was introduced and we are using custom branch with that. That locking renders access very slow. As in many use cases network access could be almost as fast as the drive access or even faster (e.g. on an EC2 instance), it might make sense to make such caching optional, or even not enabled by default at all so we could use non-patched fsspec with default installation of datalad-fuse.
So let's add
--caching none,ondiskwithnoneas default, and provide documentation in e.g. README to say that patched version of fsspec is needed forondisk.NB while coming up with interface, might be worth considering that later we might make caching support configurable as well, so "ondisk" might not be descriptive enough. But we could think about it whenever we get there.