transferwise / pipelinewise-target-s3-csv

Singer.io Target for CSV on S3 - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
15 stars 43 forks source link

Add minio support (via endpoint_url) #28

Open korpa opened 4 years ago

korpa commented 4 years ago

I'd like to use minio instead of AWS S3. Sadly my Python knowhow is to bad to do it on my own. An example is provided by https://docs.min.io/docs/how-to-use-aws-sdk-for-python-with-minio-server.html

Samira-El commented 3 years ago

hey, I'm not familiar with minio but we've recently added support for aws_endpoint_url, perhaps that would work for your case?

dluftspring commented 1 year ago

@Samira-El was also looking at this I think the only thing that needs to be added is some conditional logic when you instantiate the boto3 client. I think without setting verify=False it won't work. Additionally use_ssl may also have to be disabled at the client level. Happy to try contributing the feature but not sure how you'd want it done stylistically.

boto3.resource('s3', 
    endpoint_url='<endpoint_url>',
    aws_access_key_id='<key_id>',
    aws_secret_access_key='<access_key>',
    aws_session_token=None,
    verify=False
)
dluftspring commented 1 year ago

Ignore my comment above 😄 confirmed that this actually works on latest default branch as is. You can close this issue!