search

transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
12 stars 65 forks source link

Allow boto3 to discover credentials, add `aws_redshift_copy_role_arn` #23

Closed Limess closed 4 years ago

Limess commented 5 years ago

This change allows boto3 to search for credentials automatically if an access and secret key are not provided.

This allows the target to use environment, EC2, ECS or credentials from an AWS_PROFILE file if present, rather than forcing them to be passed explicitly. Credentials will still be used by default if provided, but they are no longer required.

Added an option to explictly pass credentials for use in the COPY operation, this allows a different role arn to be used than that fetched from the credentials chain.

This requires a role to be setup for the integration tests which can be used by COPY, set to TARGET_REDSHIFT_AWS_REDSHIFT_COPY_ROLE_ARN.

Tested getting credentials locally using an ~/.aws/credentials file and AWS_PROFILE environment variable, so other mechanisms should also work.

N.B this change means that the target will fail when trying to perform S3 operations for the first time if credentials are missing, rather than on instantiation. If failing fast is preferred I believe we could validate whether keys exist in the frozen credentials object.

Fixes https://github.com/transferwise/pipelinewise-target-redshift/issues/10

koszti commented 4 years ago

this is awesome. I added some comments and requested some small changes. It caused my redshift cluster to start hanging when aws_redshift_copy_role_arn was not set but seems like the problem found and easy to fix.