search

databricks / iceberg-kafka-connect

Apache License 2.0
220 stars 49 forks source link

Connector creating table in Default schema instead of specified bucket for automatic created tables #300

Open kevingomez93 opened 1 month ago

kevingomez93 commented 1 month ago

I have configured an Iceberg Sink Connector with the following configuration:

connector.class=io.tabular.iceberg.connect.IcebergSinkConnector
iceberg.tables.evolve-schema-enabled=true
tasks.max=2
iceberg.tables.schema-override.event_date.type=Date
iceberg.catalog.client.region=us-east-1
iceberg.tables.auto-create-enabled=true
iceberg.control.commit.timeout-ms=1480000
iceberg.tables=schema.some_table
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter=org.apache.kafka.connect.storage.StringConverter
iceberg.tables.default-partition-by=event_date
topics=some_topic
iceberg.catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO
iceberg.control.commit.interval-ms=120000
iceberg.catalog.uri=thrift://some-adress:9083
value.converter.schemas.enable=false
iceberg.catalog.warehouse=s3a://streaming-bucket/warehouse
iceberg.catalog.type=hive
iceberg.catalog.s3.path-style-access=true
iceberg.catalog.s3.bucket.name=streaming-bucket

Even though the configuration explicitly specifies the iceberg.catalog.warehouse parameter and iceberg.catalog.s3.bucket.name as s3a://streaming-bucket/warehouse and the bucket name as streaming-bucket, the connector automatically creates the Iceberg table in a default bucket rather than the specified bucket and schema.

Do you know if is possible to set the bucket where the table in S3 should be placed? Is there something that I am missing in the configuration?