confluentinc / kafka-connect-jdbc

Kafka Connect connector for JDBC-compatible databases
Other
19 stars 955 forks source link

Add configuration for Keep Interval (ms) #1306

Closed ITman1 closed 1 year ago

ITman1 commented 1 year ago

Problem

We do use timestamp+incrementing mode, we have unique incrementing column with every change, but we still need timestamp column for better performance, as we cannot create index on incrementing column because it is only pseudo column - ORA_ROWSCN . Our transactions are having quite long transaction timeout and we would like to have near realtime streaming therefore we cannot use already existing timestamp.delay.interval.ms property. Timestamp column has instant when change happened, but incrementing column is associated with the instant when transaction completed.

Solution

Extend WHERE condition for timestamp column into the past based on configuration property timestamp.keep.interval.ms. Applicable only in timestamp+incrementing mode.

Does this solution apply anywhere else?

Test Strategy

Testing done:

Release Plan

CLAassistant commented 1 year ago

CLA assistant check
All committers have signed the CLA.