search

tabular-io / iceberg-kafka-connect

Apache License 2.0
203 stars 46 forks source link

Iceberg Sink property iceberg.tables.upsert-mode-enabled=true not working #180

Closed rj-u-developer closed 8 months ago

rj-u-developer commented 8 months ago

It is throwing below error when I set iceberg.tables.upsert-mode-enabled=true to update the existing data in iceberg table:-

Task is being killed and will not recover until manually restarted. Error: Equality field ids shouldn't be null or empty when creating equality-delete writer (org.apache.kafka.connect.runtime.WorkerSinkTask:612) ava.lang.IllegalStateException: Equality field ids shouldn't be null or empty when creating equality-delete writer

ajantha-bhat commented 8 months ago

Try setting iceberg.tables.default-id-columns with a column name from upsert key.

https://github.com/tabular-io/iceberg-kafka-connect?tab=readme-ov-file#configuration

rj-u-developer commented 8 months ago

@ajantha-bhat , I already have set the above property. Please see sink properties below which I am using:-

connector.class=io.tabular.iceberg.connect.IcebergSinkConnector iceberg.catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog iceberg.tables.schema-force-optional=true topics=MY_TOPIC_NAME tasks.max=1 iceberg.catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO iceberg.catalog.client.region=REGION_NAME iceberg.tables.upsert-mode-enabled=true iceberg.table.default.iceberg.id-columns=id iceberg.tables.auto-create-enabled=true iceberg.tables=ICEBERG_DB.TABLE_NAME value.converter.schemas.enable=true iceberg.catalog.warehouse=s3://MY_BUCKET_NAME/DIR1 auto.create=true value.converter=org.apache.kafka.connect.json.JsonConverter key.converter=org.apache.kafka.connect.storage.StringConverter

ajantha-bhat commented 8 months ago

@bryanck: Thoughts?

rj-u-developer commented 8 months ago

@ajantha-bhat @bryanck , let me know if you need any further details.

bryanck commented 8 months ago

Are you using the latest version? There was an issue that was fixed in this PR a while back.

bryanck commented 8 months ago

Also, the property name is iceberg.tables.default-id-columns. You have iceberg.table.default.iceberg.id-columns.

rj-u-developer commented 8 months ago

@bryanck @ajantha-bhat

I changed the property name to iceberg.tables.default-id-columns and it is working fine now.

Thank you very much! :)

mattssll commented 8 months ago

could someone please close this issue then, can you do it @big-data-developer ?

rj-u-developer commented 8 months ago

done.