oruebel
commented
3 months ago PyNWB has a tutorial for the deafult/basic options: https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html#streaming-method-1-fsspec
https://github.com/NeurodataWithoutBorders/nwb_benchmarks/pull/43 implements this, but it does not implement other caching options for fsspec.
Fsspec offers a large number of cache options
Our current test is only for the non-cache behavior
PyNWB has a tutorial for the deafult/basic options: https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html#streaming-method-1-fsspec
But fsspec also has MANY other cache options and read-ahead styles: https://github.com/fsspec/filesystem_spec/blob/master/fsspec/caching.py#L72
We should push those to their limits in our tests to see what works best (especially in cases of repeated access to contents)