search

tabular-io / iceberg-kafka-connect

Apache License 2.0
202 stars 46 forks source link

Access kafka cluster topic in confluent cloud #281

Open arshahmad1 opened 2 months ago

arshahmad1 commented 2 months ago

I'm trying to connect this connector to the kafka topic present in my confluent cloud. I was able to access the confluent kafka topic from an ec2 instance. Now I'm creating this connector in same VPC, subnet and attaching same security group as of ec2 instance but I'm unable to get the data. These are the configurations that I'm using but I'm getting timeout 

iceberg.kafka.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="<CONFLUENT_API_KEY>" password="<CONFLUENT_API_SECRET>";
iceberg.kafka.security.protocol=SASL_SSL
iceberg.kafka.sasl.mechanism=PLAIN

connector.class=io.tabular.iceberg.connect.IcebergSinkConnector
table.write-format=parquet
iceberg.tables.evolve-schema-enabled=true
iceberg.fs.s3a.path.style.access=true
table.namespace=pocckafkaflink2
iceberg.catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog
tasks.max=1
topics=confluent_test_topic2
iceberg.catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
iceberg.catalog.client.region=us-east-1
iceberg.fs.s3a.aws.credentials.provider=com.amazonaws.auth.DefaultAWSCredentialsProviderChain
iceberg.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
iceberg.tables=pocckafkaflink2.mskmytopicmaib
value.converter.schemas.enable=false
iceberg.catalog.warehouse=s3://bucket/data4
value.converter=org.apache.kafka.connect.json.JsonConverter
table.auto-create=true
arshahmad1 commented 2 months ago

is this the correct format to provide sasl.jaas.config, security.protocol and sasl.mechanism