search

RedHatInsights / expandjsonsmt

Kafka Connect SMT to expand JSON field
Apache License 2.0
17 stars 18 forks source link

Drop tombstones #2

Closed mhaagens closed 4 years ago

mhaagens commented 4 years ago

"transforms.unwrap.drop.tombstones": "false" does not work with this SMT. Does it not produce null values for the fields it affects?

Josca commented 4 years ago

@mhaagens sorry I do not know how does it work and I did not need it. So probably not if you tested that. What is it good for?

mhaagens commented 4 years ago

@Josca It lets connectors (ex. ElasticsearchSinkConnector) delete documents based on "tombstone" records from a source (Debezium in my case). So if I delete a row in Postgres the corresponding document in Elasticsearch is also deleted. As I understand it it needs a null record to work, using: "behavior.on.null.values": "delete". Ref; https://debezium.io/documentation/reference/0.10/configuration/event-flattening.html#_record_filtering_for_delete_records

Any chance of supporting this?

Josca commented 4 years ago

@mhaagens yes I was also solving the way how to synchronize deleting of records. Finally I solved this by adding __deleted flag to source connector (Postgres) adding "transforms.unwrap.delete.handling.mode": "rewrite" option to unwrap transformation. And then I created another SMT to transform records with false __deleted field - to null. I added this SMT to Elastic sing connector. In combination with option "behavior.on.null.values":"delete" it works as I needed. Maybe it's a little complicated, but it works. I've tried multiple ways, including tombstones one, but it did not work for my setup.

mhaagens commented 4 years ago

Ah, gotcha. Thanks for posting the link, I'll give the other SMT a shot!