search

databricks / iceberg-kafka-connect

Apache License 2.0
220 stars 49 forks source link

Creating Iceberg Table in snowflake using Glue Catalog #266

Open sharpsoul opened 5 months ago

sharpsoul commented 5 months ago

Hi All,

I'm using Kafka connector to Read CDC data sent by Postgres Source connector. I'm able to create Iceberg table using Glue catalog and I'm able to query it using Athena.

Now when I try to create Table in snowflake using same table from AWS Glue it gives me error "Iceberg tables using row-level deletes are not supported."

Snowflake Says they don't support external catalog with row-level-delete. In the following documentation they also talk about Copy-on-write (default) and Merge-on-read. Snowflake Document

I tried setting up tables write properties to copy-on-write, and it is working. I used following config

iceberg.tables.auto-create-props.write.merge.mode=copy-on-write
iceberg.tables.auto-create-props.write.update.mode=copy-on-write
iceberg.tables.auto-create-props.write.delete.mode=copy-on-write

But I still get and error about "Iceberg tables using row-level deletes are not supported" in Snowflake

Is there any way to create compatible iceberg table which we can attach to snowflake.

Thanks in advance.

Update

I tried using version 1 of iceberg but in AWS Glue I can see table created but no data even though there are data files in S3 bucket under data folder.