search

tabular-io / iceberg-kafka-connect

Apache License 2.0
192 stars 41 forks source link

Partitioned table not created when auto created is enabled #123

Closed haripriyarhp closed 10 months ago

haripriyarhp commented 10 months ago

Hi, I have given the below configs. But the table that is auto created is not partitioned even if the partition-by column is mentioned. Only a normal table is created. iceberg.catalog.catalog-impl: org.apache.iceberg.aws.glue.GlueCatalog iceberg.catalog.io-impl: org.apache.iceberg.aws.s3.S3FileIO iceberg.catalog.s3.staging-dir: /opt/kafka/staging iceberg.catalog.warehouse: s3://bucket-name/iceberg iceberg.control.commit.interval-ms: 30000 iceberg.tables: databasename.tablename iceberg.table.databasename.tablename.partition-by: id iceberg.tables.auto-create-enabled: true key.converter: org.apache.kafka.connect.storage.StringConverter topics: topic-name value.converter: io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url: http://localhost:8081 value.converter.schemas.enable: true value.converter.value.subject.name.strategy: io.confluent.kafka.serializers.subject.RecordNameStrategy

bryanck commented 10 months ago

I tested your config and a Glue table was created with partitioning. Does your message have a field named id?

haripriyarhp commented 10 months ago

Yes, there is a column id in the message and it is also visible in table . But the DDL of the table created in Athena is as below

CREATE TABLE databasename.tablename ( id string, origin string, ... ... dt string) LOCATION 's3://bucket-name/iceberg/databasename.db/tablename TBLPROPERTIES ( 'table_type'='iceberg', 'write_compression'='zstd' );

haripriyarhp commented 10 months ago

Also in the glue catalog, there is no partition created and also not in the metadata json created for the iceberg.

bryanck commented 10 months ago

You may want to check the integration tests and see if those provide any insight. I confirmed the metadata files in my test contained the partition spec.

bryanck commented 10 months ago

My only other thought is that the config isn’t getting picked up for the table, if there is a slight mismatch in the name for example.

haripriyarhp commented 10 months ago

Hmm... thanks for the prompt reply. Will check the configs once again.

bryanck commented 10 months ago

I'm going to close this, feel free to reopen if you have a way to reproduce and I'll look into it.