This wouldn’t be that hard to add, but there are two problems that I see:
The maximum number of items returned from an S3 list_objects query is 1,000 items, requiring pagination to return all items. This would potentially result in some differences in the way that the different adapters work (the Disk adapter would not have that limitation) unless some sort of pagination mechanism was always assumed. The ideal here would be to make things work like FileStore.Stat.
The S3 list_objects query acts more like find x -type f than ls, in that “subdirectories” are returned as part of the search because the key space in S3 is flat, and the prefix is literally that: a prefix (but the / still acts a little special).
This wouldn’t be that hard to add, but there are two problems that I see:
The maximum number of items returned from an S3
list_objects
query is 1,000 items, requiring pagination to return all items. This would potentially result in some differences in the way that the different adapters work (the Disk adapter would not have that limitation) unless some sort of pagination mechanism was always assumed. The ideal here would be to make things work likeFileStore.Stat
.The S3
list_objects
query acts more likefind x -type f
thanls
, in that “subdirectories” are returned as part of the search because the key space in S3 is flat, and theprefix
is literally that: a prefix (but the/
still acts a little special).I’d be happy to contribute this.