Closed haripriyarhp closed 10 months ago
I tested your config and a Glue table was created with partitioning. Does your message have a field named id
?
Yes, there is a column id in the message and it is also visible in table . But the DDL of the table created in Athena is as below
CREATE TABLE databasename.tablename ( id string, origin string, ... ... dt string) LOCATION 's3://bucket-name/iceberg/databasename.db/tablename TBLPROPERTIES ( 'table_type'='iceberg', 'write_compression'='zstd' );
Also in the glue catalog, there is no partition created and also not in the metadata json created for the iceberg.
You may want to check the integration tests and see if those provide any insight. I confirmed the metadata files in my test contained the partition spec.
My only other thought is that the config isn’t getting picked up for the table, if there is a slight mismatch in the name for example.
Hmm... thanks for the prompt reply. Will check the configs once again.
I'm going to close this, feel free to reopen if you have a way to reproduce and I'll look into it.
Hi, I have given the below configs. But the table that is auto created is not partitioned even if the partition-by column is mentioned. Only a normal table is created. iceberg.catalog.catalog-impl: org.apache.iceberg.aws.glue.GlueCatalog iceberg.catalog.io-impl: org.apache.iceberg.aws.s3.S3FileIO iceberg.catalog.s3.staging-dir: /opt/kafka/staging iceberg.catalog.warehouse: s3://bucket-name/iceberg iceberg.control.commit.interval-ms: 30000 iceberg.tables: databasename.tablename iceberg.table.databasename.tablename.partition-by: id iceberg.tables.auto-create-enabled: true key.converter: org.apache.kafka.connect.storage.StringConverter topics: topic-name value.converter: io.confluent.connect.avro.AvroConverter value.converter.schema.registry.url: http://localhost:8081 value.converter.schemas.enable: true value.converter.value.subject.name.strategy: io.confluent.kafka.serializers.subject.RecordNameStrategy