search

tabular-io / iceberg-kafka-connect

Apache License 2.0
202 stars 46 forks source link

Glue configurations for iceberg connector #264

Open arshahmad1 opened 3 months ago

arshahmad1 commented 3 months ago

Hi Team, I'm trying to configure this connector on confluent cloud, can someone please guide me how can I provide the configurations for connector to connect to aws glue as catalog? I already went through the Iceberg glue catalog documentation but I'm there's no related configuration there. I've a IAM role created in AWS with the required access to S3 and Glue catalog but I can't find any configuration that I can use to link the connector with aws glue. I've already tried iceberg.catalog.client.assume-role.arn and client.assume-role.arn.

Also there's no configuration to provide the glue catalog database and table names for the data to land. Can anyone please take a loot at it and help me with this 🙂
Thanks!

arshahmad1 commented 3 months ago

Here's my current configurations

{ "topics": "kafka_topic_name", "iceberg.tables": "s3_bucket_name.s3_folder_name", [# Not sure about this property]() "iceberg.catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog", "iceberg.catalog.warehouse": "s3://s3_bucket_name/s3_folder_name", "iceberg.catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO", "iceberg.catalog.client.assume-role.arn": "arn:aws:iam:::role/", "client.assume-role.arn": "arn:aws:iam:::role/", "value.converter.schemas.enable": "false", "value.converter": "org.apache.kafka.connect.json.JsonConverter", "key.converter": "org.apache.kafka.connect.storage.StringConverter" }

arshahmad1 commented 3 months ago

hey @tabmatfournier, Sorry to ping you directly. Can you please help me here.

sharpsoul commented 3 months ago

You need to set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as env variables but I know you can't do that in confluent. Look for an option to set AWS configurations in confluent.

arshahmad1 commented 3 months ago

Thanks @sharpsoul, really appreciate your help. Let me check that.

braislchao commented 1 month ago

Hey @arshahmad1 , have you found a working config? Thanks in advance.

braislchao commented 1 month ago

I'm trying with this config properties for AWS access on a custom Confluent Cloud connector: 

"iceberg.catalog.s3.access-key-id": "***********",
"iceberg.catalog.s3.secret-access-key": "**********",

However im facing issues with the connector Transactional Id producer, even though I'm using an API key with full permissions over the cluster.

Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted Transactional Id authorization failed.

Have you tried those properties? Did you manage to get the connector running?

Thanks!