scylladb / kafka-connect-scylladb

Kafka Connect Scylladb Sink
Apache License 2.0
42 stars 22 forks source link

delivery semantics #40

Closed raphaelauv closed 3 years ago

raphaelauv commented 3 years ago

Hi , is the connector a best-effort connector ( accept lost and duplicate of messages from kafka to scylladb )

or is at-least-once, offsets of the consumer are only commits when data is asserted to arrive in scylladb ( every msg in kafka will be at least present 1 time in scylladb ) ?

or best case but expensive exactly-once connector.

Thank you

avelanarius commented 3 years ago

The connector is a at-least-once connector. In case of failure to write to Scylla, the write will be retried. It is NOT exactly-once.

In case of the connector receiving invalid messages on the source topic (for example if you configured it with JSON converter, but sent a non-JSON message), the connector has configurable policy by behavior.on.error:

raphaelauv commented 3 years ago

Thank you so much , that is perfectly clear !