tabular-io / iceberg-kafka-connect

Apache License 2.0
169 stars 31 forks source link

Provide AWS access key ID and secret access key through configuration properties #249

Open prakash-42 opened 1 month ago

prakash-42 commented 1 month ago

I've been trying to setup the iceber-sink-connector in confluent cloud with glue catalog, which requires me to configure AWS access key and secret access key. Is it possible to provide these credentials through some configuration property? I tried iceberg.catalog.s3.access-key-id and iceberg.catalog.s3.secret-access-key but it didn't work for glue catalog. Here are my configuration properties for the connector:

 "iceberg.tables": "<database>.<table>",
 "iceberg.tables.auto-create-enabled": "true",
 "iceberg.tables.evolve-schema-enabled": "true",
 "iceberg.catalog.client.region": "ap-south-1",
 "iceberg.catalog.s3.access-key-id": "xxxxxxxxxxxx",
 "iceberg.catalog.s3.secret-access-key": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
 "iceberg.catalog.warehouse": "s3://<bucket_name>/<folder_path>",
 "iceberg.catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog",

Setting up the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY would probably solve my issue, but I don't have the option to set it as I'm running this connector as a custom connector in confluent cloud. I also do not get control over the connect workers that run this connector. How can I provide the AWS credentials?

Any help on this is much appreciated, thank you!