databricks / iceberg-kafka-connect

Apache License 2.0
220 stars 49 forks source link

MSK Connector with tasks.max > 1 throws an AlreadyExistsException while creating glue table #270

Closed sagarm-traveloka closed 4 months ago

sagarm-traveloka commented 4 months ago

We have defined a kafka cluster with 3 partitions count. We referred https://tabular.io/blog/streaming-event-data-to-iceberg-with-kafka-connect/ and decided to consume the data with high speed and for that set tasks.max=3.

Below is the exception we noticed -> org.apache.iceberg.exceptions.AlreadyExistsException: Cannot commit iceberg.ledgers because its Glue table already exists when trying to create one at org.apache.iceberg.BaseMetastoreCatalog$BaseMetastoreCatalogTableBuilder.create(BaseMetastoreCatalog.java:199) at io.tabular.iceberg.connect.data.IcebergWriterFactory.lambda$autoCreateTable$1(IcebergWriterFactory.java:106) Could you please suggest any alternate way to tackle this case?