transferwise / pipelinewise-target-redshift

Singer.io Target for Amazon Redshift - PipelineWise compatible

https://transferwise.github.io/pipelinewise/

Other

12 stars 65 forks source link

ECS credentials not being correctly picked up by boto3 when no keys are provided. #36

Closed koszti closed 4 years ago

koszti commented 4 years ago

31 meant to fix it but looks like this still exists. Futher details in #31

paul-english commented 4 years ago

I just tried updating boto & using the same s3 client initialization from https://github.com/transferwise/pipelinewise/pull/285, neither worked. I'm still seeing an InvalidAccessKeyId exception.

Pipelinewise in it's docker container works with s3 for fastsync into redshift, but doing key based replication doesn't work (since it uses this tap via singer). Additionally trying to use this target directly as a singer pipe causes the same error outside of any docker container.

paul-english commented 4 years ago

I think I've figured out the source of this error, https://github.com/transferwise/pipelinewise-target-redshift/blob/master/target_redshift/db_sync.py#L236-L237

If I move these into the conditional so these lines are only set when we have a key & secret I'm able to use the instance role.

koszti commented 4 years ago

cool, sounds great that you could make it work. Can you please send a small PR so we can review, test and release a new package?

paul-english commented 4 years ago

@koszti https://github.com/transferwise/pipelinewise-target-redshift/pull/38

koszti commented 4 years ago

thanks for this. It solved the problem but looks like it's causing another scenario to fail. Further details in #38

koszti commented 4 years ago

fixed by https://github.com/transferwise/pipelinewise-target-redshift/pull/38