Does S3 iter_bucket function support custom s3 endpoint_url?

piskvorky / smart_open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

MIT License

3.2k stars 384 forks source link

Does S3 iter_bucket function support custom s3 endpoint_url? #740

Open phatvo2015 opened 1 year ago

phatvo2015 commented 1 year ago

Hi! Is there anyway to pass custom s3 endpoint url for module s3's function iter_bucket. As I see, so far it only allows to pass session kwargs - https://github.com/RaRe-Technologies/smart_open/blob/develop/smart_open/s3.py#L1107 . It would be nice if endpoint url is supported so that we can test against our mock S3 bucket server.

gjedlicska commented 1 year ago

Plus one for this feature. It would be nice to be able to pass in the client / resource kwargs along the session kwargs.

5t33 commented 11 months ago

For anyone stuck on this thread, Boto3 now supports environment variable AWS_ENDPOINT_URL_S3 as of 1.28.0. This worked as a workaround for me using localstack s3.