search

SAP / kafka-connect-sap

Kafka Connect SAP is a set of connectors, using the Apache Kafka Connect framework for reliably connecting Kafka with SAP systems
Apache License 2.0
122 stars 56 forks source link

Polling mode based on timestamp #100

Closed srkpers closed 2 years ago

srkpers commented 2 years ago

I was trying to do polling based on timestamp (timestamp.column.name) and got an error. I then checked the documentation and looks like only "bulk" and "incrementing" are supported at this point. Any idea when "timestamp" based polling to detect changes will be supported? This will be really helpful as we do not have any HANA table with strictly incrementing values for any column which leaves us with only one option to stream data which is "query" based. With query based approach we will not be able to get data into Kafka in near real time.

elakito commented 2 years ago

You can set the incrementing column to your timestamp column. If the incoming data are stored between two polling cycles with different timestamps, the polling should work and all data should get fetched. If the timestamp of the first record in one cycle is identical to the timestamp of the last record in its previous cycle, those records will be excluded from getting fetched. But this problem can't be avoided if there is no unique column value to filter out the duplicates.

elakito commented 2 years ago

It's been more than a month since last activity, so I suppose we can close this ticket.