search

rzane / file_store

🗄️ A unified interface for file storage backends
MIT License
19 stars 3 forks source link

Interest in a `list_files` or `list_objects` function? #4

Closed halostatue closed 3 years ago

halostatue commented 4 years ago

This wouldn’t be that hard to add, but there are two problems that I see:

  1. The maximum number of items returned from an S3 list_objects query is 1,000 items, requiring pagination to return all items. This would potentially result in some differences in the way that the different adapters work (the Disk adapter would not have that limitation) unless some sort of pagination mechanism was always assumed. The ideal here would be to make things work like FileStore.Stat.

  2. The S3 list_objects query acts more like find x -type f than ls, in that “subdirectories” are returned as part of the search because the key space in S3 is flat, and the prefix is literally that: a prefix (but the / still acts a little special).

I’d be happy to contribute this.

rzane commented 3 years ago

This would be a welcome addition!