search

confluentinc / kafka-connect-jdbc

Kafka Connect connector for JDBC-compatible databases
Other
19 stars 955 forks source link

Support for Exactly once semantics for JDBC Source connectors (with release of KIP-618) #1316

Open aditya81070 opened 1 year ago

aditya81070 commented 1 year ago

Hey team, thanks for creating this connector. We are using this connector from a long time and always worried about data duplication because of reasons mentioned here

We just found out that KIP-618 is in almost completing phase (almost everything in 3.3.0) and it allows EOS supports for source connectors natively.

What are the plans to enable this for this connector? I can see that some of the connectors already started adopting this (check list here

Why we need this?

We are loading very heavy data (like everyone else) using JDBC source connector and we have tried producer idempotency as well as transactional producers but still data is duplicated because of records + offset commit functioning in different thread.

mortezaalizadeh commented 8 months ago

Hi team, is this currently released or we are still waiting for "Exactly once semantics" capability to be implemented? cheers

Golfer474 commented 6 months ago

Hi, wanted to check in on this as we use this connector and could benefit from EOS. What is the work effort to enable EOS for the JDBC Connector? If small, I could potentially open a PR for it.

treinoso commented 1 month ago

Is there any update on this? Thanks!