PelicanPlatform / pelicanfs

An fsspec implementation that uses the pelican client
https://pelicanplatform.org/
Apache License 2.0
1 stars 4 forks source link

Reduce director calls for get_dirlist_url #47

Open turetske opened 6 months ago

turetske commented 6 months ago

We have quite a few calls to the listing url that sometimes occur within the same overall call.

Such as when glob does a find on its allpaths list. When this occurs, it gets the goes through the call to the get the dirlist url. multiple times in order to run a find on all of its paths. We can assume that subdirectories of these have the same dirlist url.

I suggest implementing a caching of a dirlist url the same way we do with the cache. If the prefix matches, then try the same dirlist url rather than pulling a new one from the director.