search

databricks / iceberg-kafka-connect

Apache License 2.0
219 stars 49 forks source link

[DebeziumTransform] Allow _cdc.ts to be derived from source.ts_ms rather than <root>.ts_ms. #189

Open megri opened 9 months ago

megri commented 9 months ago

We are relying on the nested field source.ts_ms to track creation/update times where the source table doesn't have or misuses corresponding columns. It would be great if the SMT could be configured to support this, or better yet if both timestamps were kept around.

braislchao commented 3 months ago

I stumbled in this issue too, examining the SMT code it should be very easy to expose the fields you want with simple changes there:

https://github.com/tabular-io/iceberg-kafka-connect/blob/main/kafka-connect-transforms/src/main/java/io/tabular/iceberg/connect/transforms/DebeziumTransform.java