search

thanos-io / objstore

Go module providing unified interface and efficient clients to work with various object storage providers until like GCS, S3, Azure, SWIFT, COS and more.
Apache License 2.0
101 stars 72 forks source link

Pagination Support #80

Open gotjosh opened 1 year ago

gotjosh commented 1 year ago

Some providers, such as:

s3

Support parameters such as max-keys or start-after. More in the API reference

gc

Support parameters such as maxResults, pageToken and startOffset. More in the API reference

Have some support that allows us to paginate results when listing items.

Is there room/need for having the client support these? In our case, we have a need to allow client-facing APIs to iterate over 150k keys at a time and as you can expect the experience of having to iterate through all of these keys at once it's not ideal.

In reality, I don't have much data on timings hence filing #79 but I reckon I'd like to get the conversation started on wether this is something we would like to support.

To begin with, I'm open to the idea of having us support this for the three major cloud providers: GCS, AWS and Azure.

yeya24 commented 11 months ago

I am open to support those parameters! PR is welcomed.