search

getindata / kafka-connect-iceberg-sink

Apache License 2.0
76 stars 27 forks source link

How to deal with TOAST values? #17

Open ndrluis opened 1 year ago

ndrluis commented 1 year ago

Hello,

I'm using this connector with postgresql debezium and we have some UPDATE events with the TOAST placeholder, it's possible to deal with TOAST values at consumer level excluding unchanged values?

https://debezium.io/blog/2019/10/08/handling-unchanged-postgres-toast-values/

gliter commented 1 year ago

This is not currently supported. I think you could write Kafka Connect transofrmation to discard entire event.

daniel-suissa commented 1 year ago

If you discard the entire event, wouldn't changes to other columns in the same UPDATE statement be discarded as well?