search

memiiso / debezium-server-iceberg

Replicates any database (CDC events) to Apache Iceberg (To Cloud Storage)
Apache License 2.0
171 stars 35 forks source link

Test and enable consuming without event flattening #339

Closed ismailsimsek closed 2 weeks ago

ismailsimsek commented 1 month ago

After https://github.com/memiiso/debezium-server-iceberg/pull/232 and https://github.com/memiiso/debezium-server-iceberg/pull/233 it should be possible to run the consumer without unwrap

debezium.transforms=unwrap

Test and enable consuming without event flattening, currently its required configuration.

ismailsimsek commented 2 weeks ago

Depends on #342

ismailsimsek commented 2 weeks ago

Implemented with https://github.com/memiiso/debezium-server-iceberg/issues/342

Documented with #357

Without event flattening iceberg consumer can only run with append mode. Without event flattening upsert mode and creation of identifier fields are not supported. 

debezium.sink.iceberg.upsert=false
debezium.sink.iceberg.create-identifier-fields=false