search

transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
12 stars 65 forks source link

Allow STS token to be used #14

Closed Limess closed 4 years ago

Limess commented 5 years ago

This allows a workaround for temporary credential usage, which gives the possibility of using IAM roles via STS.

The example use case here is using Apache Airflow: I wish to use the built in credentials providers which airflow uses - to do this I'm using IAM roles which are assumed by boto3 automatically, and then taking the access key/secret key/sts token and providing them to libraries which need them.

It'd be equally useful to allow an AWS profile or role to be used in future, specifically to separate the role used by Redshift COPY and the S3 credentials to do the multipart upload.

Tested against our redshift environment.

See equivalent PR for the other open source redshift target https://github.com/datamill-co/target-redshift/pull/31

koszti commented 4 years ago

You says this was originally a workaround but am I right this is still relevant and useful once #23 got merged? Looks like Yes, but please confirm how do you feel about it?

Limess commented 4 years ago

@koszti yes, this is still useful to us/generally

koszti commented 4 years ago

@Limess can you please resolve conflicts and it will get merged.

Limess commented 4 years ago

Great! I've rebased this now.