Listing s3 paths with objects exeeding 1000 items

[nadavgut]

Hi there! I wondered if S3path has a support for the use case of getting objects in numbers of above 1000 under a certain key in s3?

Such as described here and suggested to use boto3's list_v2_objects paginator.

Thanks in advance!

[liormizr]

Hi @nadavgut

You can try with our configuration setup, documentation here.

Basically it will look something like this:

from s3path import S3Path, register_configuration_parameter
page_size = 2_000
folder = S3Path('/<bucket>/folder/')

register_configuration_parameter(folder, parameters={'MaxKeys': page_size})

Please let me know if that helps

Thanks

[liormizr]

@nadavgut did my answer helped you?

[nadavgut]

@liormizr , thank you for your reply. Theoretically, I can do a tests using localstack or in some Qa environment using real bucket with more than 1000 objects, but frankly, I didn't have time to check it.

Have you managed to test above suggestion by adding some test to project's tests you probably already have? I think it would be the best for that.

[liormizr]

Hi @nadavgut

I know that my suggestion works :smile: From the configuration doc you can see that every method in boto3 that get MaxKeys will now get this param.

I'm asking you because you opened this issue and I want to close it if we don't need to do nothing here...

[nadavgut]

Cool, so the tests are on boto3 api 😉 thanks for you work in developing this nice package

בתאריך יום א׳, 24 בדצמ׳ 2023, 09:17, מאת liormizr @.***

:

Hi @nadavgut https://github.com/nadavgut

I know that my suggestion works 😄 From the configuration doc https://github.com/liormizr/s3path/blob/master/docs/advance.rst you can see that every method in boto3 that get MaxKeys will now get this param.

I'm asking you because you opened this issue and I want to close it if we don't need to do nothing here...

— Reply to this email directly, view it on GitHub https://github.com/liormizr/s3path/issues/151#issuecomment-1868450404, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQB25B4IQE6Y6NYQVYJJN3YK7JHDAVCNFSM6AAAAAA6WZBAFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRYGQ2TANBQGQ . You are receiving this because you were mentioned.Message ID: @.***>