Now suppose there’s a case where connector faced some errors while publishing the records from this response and it threw an error, the error handling mechanism would attempt a retry on the from_op_id but this time the tablet safe hybrid time value would be higher thus resulting in filtering of some records which would cause data loss.
Solution
This diff moves the tablet safe time updation after the records have been processed successfully by the connector.
Problem
In the connector flow, we call
GetChangesand then immediately set the tablet safe hybrid time from the response i.e. https://github.com/yugabyte/debezium-connector-yugabytedb/blob/main/src/main/java/io/debezium/connector/yugabytedb/YugabyteDBStreamingChangeEventSource.java#L533Now suppose there’s a case where connector faced some errors while publishing the records from this response and it threw an error, the error handling mechanism would attempt a retry on the
from_op_idbut this time the tablet safe hybrid time value would be higher thus resulting in filtering of some records which would cause data loss.Solution
This diff moves the tablet safe time updation after the records have been processed successfully by the connector.
Closes yugabyte/yugabyte-db#21940