search

nteract / commuter

🚎 Notebook sharing hub
BSD 3-Clause "New" or "Revised" License
495 stars 66 forks source link

support s3 directories with more than 1000 keys #314

Open andrewresnikoff opened 3 years ago

andrewresnikoff commented 3 years ago

Each list-objects-v2 API call only returns 1000 keys at a time as per the AWS Docs.

Returns some or all (up to 1,000) of the objects in a bucket.

When there are more than 1000 keys, we get back NextContinuationToken

NextContinuationToken is sent when isTruncated is true, which means there are more keys in the bucket that can be listed. The next list requests to Amazon S3 can be continued with this NextContinuationToken . NextContinuationToken is obfuscated and is not a real key

to paginate the remaining results.