search

lericson / simples3

Simple, quick Amazon AWS S3 interface in Python
BSD 2-Clause "Simplified" License
96 stars 36 forks source link

listdir limited to 1000 entries #21

Open SnapshotCiTy opened 7 years ago

SnapshotCiTy commented 7 years ago

Simple S3 is liiting the output to 1000 file when using listdir.

http://stackoverflow.com/questions/40331681/python-simples3-listdir-limit-the-output-to-1000-files

Any way / plan to make it support more than 1000 - or an unlimited number of entries?

lericson commented 7 years ago

I'll copy my response here for posterity

Author of simples3 here. I'm sorry this stopped working, but that's really on Amazon. As you can see in the implementation of S3Bucket.listdir there is support for pagination and it should be seamless. If it isn't, then that's because Amazon stopped returning the truncated flag or stopped returning markers to perform pagination.

You're welcome to do the job, but I have to question Amazon on this decision. Why break existing APIs?