search

piskvorky / smart_open

Utils for streaming large files (S3, HDFS, gzip, bz2...)
MIT License
3.17k stars 383 forks source link

cannot pass custom transport_params to s3_iter_bucket api #346

Open indrajeet307 opened 5 years ago

indrajeet307 commented 5 years ago

I cannot pass custom transport_params to s3_iter_bucket api, It defaults to using amazon s3. I am trying smart_open with a locally deployed minio server, that supports s3 like access.

mpenkov commented 5 years ago

Yes, this is currently not supported. Are you able to make a PR?

indrajeet307 commented 4 years ago

The changes, I made, in s3.py file

  1. Add optional resource_kwargs to iter_bucket api
  2. _list_bucket also accepts optional resource_kwargs, and creates a client with or without it.

I tried writing test but moto did not allow sending endpoint_url. I am getting authentication failed while trying to push commit.

mpenkov commented 4 years ago

You won't be able to push to our repository. That's by design. You need to fork our repository, push the changes to your own fork, and then open a pull request. Then we'll review the PR and merge your changes in.

Here's a guide that describes what you need to do: https://guides.github.com/activities/forking/

Let me know if you're up for it!

mpenkov commented 4 years ago

Duplicates #259

maranqz commented 4 years ago

It's not duplication because calling client doesn't sending endpoint_url to session.client('s3', ....).

5t33 commented 9 months ago

For anyone stuck on this thread, Boto3 now https://github.com/boto/boto3/issues/1375 environment variable AWS_ENDPOINT_URL_S3 as of 1.28.0. This worked as a workaround for me using localstack s3.