birdiecare / connect-smts

Kafka Connect SMTs
MIT License
17 stars 1 forks source link

feat: Add Outbox SMTs for integration events support #29

Closed artysidorenko closed 3 years ago

artysidorenko commented 3 years ago

Explained in the docs/tests - allow sending integration events to the right topic/partition.

One topic we hadn't discussed when pairing - how to handle deletions. Debezium emits two types of records in this situation (assuming we've enabled it on the ExtractNewRecordState SMT: empty tombstones and delete-rewrite records. I am dropping the tombstone because it doesn't have enough info to partition correctly, and am using the delete-rewrite to create a tombstone out of it.

Has been tested via local kafka cluster by sending HTTP requests to matching service using the setup in this PR - https://github.com/birdiecare/matching/pull/73