PelicanPlatform / pelicanfs

An fsspec implementation that uses the pelican client
https://pelicanplatform.org/
Apache License 2.0
1 stars 4 forks source link

Allow the the get_mapper to also change caches on cache failure #43

Closed turetske closed 6 months ago

turetske commented 6 months ago

Currently the get_mapper function assigned a cache to the url and that's used for all other calls.

If that cache fails, there is no fallback for the director to switch caches. If we assign a cache url and then use a different url later due to some sort of cache failure, FSMap will complain due to the change in urls assigned.

Ideally, rather than assigning a cache at the get_mapper, it should just have the namespace path value. However, that would mean that whenever a call is made cache_discovery would happen again, which isn't ideal in terms of speed. I think we need a more complicated solution involving creating our own mapping class which inherits or composes with FSMap. FSMap contains the namespace path, which would be unchanging, but that would link to the current working cache. If that cache fails, then that linkage would update without changing the underlying FSMap.