search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.7k stars 4.03k forks source link

[destination-redshift] Pass a Role's ARN instead of keys for COPY Strategy (S3 Staging) #41031

Open assaadhjb opened 2 months ago

assaadhjb commented 2 months ago

Connector Name

destination-redshift

Connector Version

0.63.4

What step the error happened?

Configuring a new connector

Relevant information

As an Airbyte Cloud user, now that standards inserts were removed, S3 Staging have become mandatory - and by the way, I personally think it should be highlighted more in the docs - we want to have both the possibilities of passing an ARN Role instead of access ID and secret access Key.

Relevant log output

No response

Contribute

marcosmarxm commented 2 months ago

@airbytehq/destinations can someone look into this issue?

edgao commented 2 months ago

this is something we want to support! There's unfortunately some code structure issues around how we handle AWS auth that are getting in the way, which we need to solve first :/ but this is one of the things that we talk about a lot within the team. We don't have a timeline on this yet, though.

assaadhjb commented 2 months ago

Hello @edgao, I understand - prioritising the deprecation of standard inserts may have needed however to first make the (required) usage of S3 consistent with authentication and access between destination-redshift and destination-s3

assaadhjb commented 1 month ago

If anyone is impacted by this produced issue, I suggest to use TF Airbyte provider. It's an improvement to at least have keys stored in a secured TF backend (however in plain ....) and be sent through secured API calls smh. Imo, this has generated a security concern for redshift-destination as a whole because AWS access keys are not meant to be used that way in production.

adam-bloom commented 1 month ago

Could supporting IAM auth for redshift be a part of this too? Or would that be a separate enhancement? Would be great to avoid password auth for redshift as well. https://docs.aws.amazon.com/redshift/latest/mgmt/generating-user-credentials.html