fsspec / filesystem_spec

A specification that python filesystems should adhere to.
BSD 3-Clause "New" or "Revised" License
1k stars 354 forks source link

RF idea: deprecate `same_names` in favor of a more generic `layout` parameter #555

Open yarikoptic opened 3 years ago

yarikoptic commented 3 years ago

ATM CachingFileSystem has a single bool option same_names to switch layout of files from /hash to /url-filename and thus does not leave room for "improvement":

Under heavy use of the cache use having a flat tree of files (/hash or /url-filename based) could lead to a very heavy directory so filesystem could become inefficient in listing that directory etc.

With above in mind, I think it would have been nice if instead of same_names there was a layout={hash,hashtree,url_filename,url_fullpath} or alike, thus allowing users to switch to most appropriate layout depending on their use case.

martindurant commented 3 years ago

Agreed that this would be useful too. I would love for someone to implement it.