search

nodefluent / sequelize-kafka-connect

:gem: nodejs kafka connect connector for MySQL, Postgres, SQLite and MSSQL
MIT License
39 stars 10 forks source link

Does this project support fetching updates on a table? #76

Closed charithf closed 3 years ago

charithf commented 3 years ago

I'm doing a POC using this project and I need to capture all Operations on a DB table. Does this project support that now or in the future. I have managed to capture initial load and subsequent inserts.

Thank you

krystianity commented 3 years ago

Hey there, by default the goal of this project is not to fetch updates from existing db entries. To fetch a column as update you either have to increase its ID (the column that is used as origin incrementingColumnName) or reset the Kafka offset, which will cause all table columns to be reproduced (replayed) - the later requires your consumers to be idempotent, which should be a general goal any way.

charithf commented 3 years ago

Thank you for your feedback. As my stack is on Node it was natural to analyse this project. I think I need a little bit more. I will check a different connector or I may have to implement something that I need on top of kafka-connect.