search

databricks / iceberg-kafka-connect

Apache License 2.0
220 stars 48 forks source link

Provide AWS access key ID and secret access key through configuration properties #249

Open prakash-42 opened 6 months ago

prakash-42 commented 6 months ago

I've been trying to setup the iceber-sink-connector in confluent cloud with glue catalog, which requires me to configure AWS access key and secret access key. Is it possible to provide these credentials through some configuration property? I tried iceberg.catalog.s3.access-key-id and iceberg.catalog.s3.secret-access-key but it didn't work for glue catalog. Here are my configuration properties for the connector:

 "iceberg.tables": "<database>.<table>",
 "iceberg.tables.auto-create-enabled": "true",
 "iceberg.tables.evolve-schema-enabled": "true",
 "iceberg.catalog.client.region": "ap-south-1",
 "iceberg.catalog.s3.access-key-id": "xxxxxxxxxxxx",
 "iceberg.catalog.s3.secret-access-key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
 "iceberg.catalog.warehouse": "s3://<bucket_name>/<folder_path>",
 "iceberg.catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog",

Setting up the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY would probably solve my issue, but I don't have the option to set it as I'm running this connector as a custom connector in confluent cloud. I also do not get control over the connect workers that run this connector. How can I provide the AWS credentials?

Any help on this is much appreciated, thank you!

braislchao commented 3 months ago

Hi @prakash-42 , can you provide details of the error trace that you are getting?

Thanks in advance!

cspoppuppy commented 2 months ago

We also have similar issue. Iceberg sink to AWS doesn't execute use the credentials from the configuration: "iceberg.catalog.s3.access-key-id": "xxxxxxxxxxxx", "iceberg.catalog.s3.secret-access-key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",

Instead, it defaults to using the role from the server's environment.

Also, it appears that Iceberg had the capability to accept credentials from the configuration, but this functionality has marked deprecated and not for production usage? https://github.com/apache/iceberg/blob/main/aws/src/main/java/org/apache/iceberg/aws/AwsClientFactories.java#L219