seglo
commented
4 years ago A Debezium connector sounds like a great idea. I took a look at the docs you mentioned. In the "Advanced Record Consuming" section it describes creating your own implementation to handle source batches of records. If calling the RecordCommitter controls when the next batch is sent, then a Debezium Source stage would only commit to request the next batch when there's a demand request.
https://debezium.io/documentation/reference/1.0/operations/embedded.html#advanced-consuming
I don't know enough about Debezium to know how failure recovery and/or restarts would work. How do you resume from some last known position? Is the state managed in whatever state store you configure with database.history?
gabfssilva
Maatary
Short description
Implement Debezium connector
Details
Debezium is a Change Data Capture (CDC) kafka connector that integrates with several databases. Although Debezium is a kafka connector, it also works as a standalone tool: https://debezium.io/documentation/reference/1.0/operations/embedded.html
The idea here is to use Debezium as a CDC source. The only issue is understand how backpressure would work here.
I also noticied there's already a PR with a CDC connector for PostgreSQL (#891), but, be able to use CDC with other databases is a nice to have.
As I create this request, the lastest final version supports the following databases:
Apache Camel already has a component for it (actually, there are a few: one for each database). We could check on how they implement it and see if it's a viable solution for Alpakka.