search

scylladb / scylla-cdc-source-connector

A Kafka source connector capturing Scylla CDC changes
Apache License 2.0
47 stars 18 forks source link

feature request: support for postimage #8

Open hartmut-co-uk opened 3 years ago

hartmut-co-uk commented 3 years ago

As a consumer of my CDC event stream (Kafka topic), with table cdc postimages enabled, I'd like to also receive data of the postimage *_cdc_log record (cdc$operation=9).

This would allow me to fully utilise the change event for stream processing use cases.

Without the CDC postimage record included to the message to Kafka the change is lost. Enriching the record as part of stream processing not only would result in extra read operations to Scylla (network IO, latency, ..) but it is also impossible to fetch the actual point-in-time row postimage of the change event (since the row might have changed again in the meantime - or no longer exist..)

Optional: either follow the cdc setting of the (source) table in question - or have the scylla-cdc-source-connector to explicitly configure (enable/disable) processing of postimages.

Example use cases:

chiragb1994 commented 2 years ago

I also need something similar. I have enabled preimage (full) on my table and postimage (true). The postimage event is created in CDC table but not streamed to Kafka. I want only the latest state of the row, to be able to stream it into a different database.