search

SAP / kafka-connect-sap

Kafka Connect SAP is a set of connectors, using the Apache Kafka Connect framework for reliably connecting Kafka with SAP systems
Apache License 2.0
119 stars 54 forks source link

Connnector skips parts of data in incrementing mode #72

Closed DmitryTaliuk closed 3 years ago

DmitryTaliuk commented 3 years ago

I used connector to get data from SAP HANA tables in both bulkand incrementingmodes with default batch.max.rows parameter (100). Table consists more then 7400 records and in bulkmode it is transferred in the Kafka topic normally and periodically with delay. But in incrementing mode I found next issue - connector gets 100 records from the table, transfers them to topic. Then it skips 100 records, gets next 100, skips 200, gets 100, skips 300, gets 100 and so on. As result only about 1400 record is saved in the Kafka topic instead 7400. May be somebody got deals with similar problem?

elakito commented 3 years ago

@DmitryTaliuk it looks like the incremental mode is not handling the chunked fetching that is supported by the bulk mode. Let me have a look...

elakito commented 3 years ago

@DmitryTaliuk the fix has been merged to master.

DmitryTaliuk commented 3 years ago

Thank you. I'll test it and report about result.

DmitryTaliuk commented 3 years ago

Incrementing mode is working fine for now. @elakito Thank you very much again. If you have no objections, I'll close the issue.

elakito commented 3 years ago

@DmitryTaliuk thanks for the validation.