search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.4k stars 3.97k forks source link

[destination-Redshift] S3 Assumed role (instead of key-id/access-key) #33039

Open sharon-clue opened 9 months ago

sharon-clue commented 9 months ago

Connector Name

destination-redshift

Connector Version

0.6.11

What step the error happened?

Configuring a new connector

Relevant information

I'm trying to setup the connector while using the S3 Staging option. the aws key-id and access are mandatory and it's not required since my my implementation is using a service account that has the needed permissions to the s3 bucket.

please change it to allow using the s3 staging option without the aws key-id and access key

Relevant log output

No response

Contribute

pquadri commented 9 months ago

same issue here, i think it's not only a matter of removing the required fields as the COPY command is explicitly using credentials (i guess it would be needed to add the service account name there)

sharon-clue commented 9 months ago

the service account is attached to the pod, so no need to do anything besides allowing the whole thing work without having the secret/access key. If i have time i'll change it and create a pull request

hvignolo87 commented 4 months ago

Same issue here. Do you have an estimated timeframe for this? I would love to help, but I know nothing about Java 😩

alexlopes commented 3 months ago

I'm facing the same problem using k8s service accounts only. Looking here I think this is not possible, because the function expects a access/secret key environment variables.

I'm getting this error

Stack Trace: java.util.NoSuchElementException: Key AWS_ACCESS_KEY_ID is missing in the map.
        at kotlin.collections.MapsKt__MapWithDefaultKt.getOrImplicitDefaultNullable(MapWithDefault.kt:24)
        at kotlin.collections.MapsKt__MapsKt.getValue(Maps.kt:360)
        at io.airbyte.cdk.integrations.destination.s3.credential.S3AssumeRoleCredentialConfig$Companion.getCredentialProvider(S3AssumeRoleCredentialConfig.kt:65)

@stephane-airbyte can you confirm please?

evantahler commented 2 months ago

For the moment, we don't have answers to these questions, as we are focused on enabling this feature for users of Airbyte Cloud. We will later explore if/how we want to enable this for self-hosted users. https://github.com/airbytehq/airbyte/pull/39371

alexlopes commented 2 months ago

Thanks for the clarification @evantahler

assaadhjb commented 2 months ago

Hello @evantahler, I can see that you are tackling this for S3 destination, however the same applies to Redshift destination with COPY strategy (S3 Staging), it makes sense enabling passing ARN Role for both, WDYT? Especially that we only have that option now that standard inserts have been removed

jacoblElementor commented 3 weeks ago

@evantahler I see that your PR is closed in favor of this PR https://github.com/airbytehq/airbyte/pull/39468 but the new PR is still open. Is the connector with Role ARN authentication still not available for Airbyte OSS? Is it available for Airbyte Enterprise?