cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.13k stars 3.81k forks source link

changefeedccl: implement protected timestamps for incremental exports #96006

Open amruss opened 1 year ago

amruss commented 1 year ago

As of 23.1, we will offer incremental exports with CDC Transformations + scheduled changefeeds. Unfortunately, we will not protect the data between exports, so you will miss messages. We should have an option to protect this data.

See scheduled backups for prior art:

Jira issue: CRDB-23892

Epic CRDB-24867

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/cdc

jayshrivastava commented 1 year ago

At the moment, all we allow is an initial scan on the table. We also run the same changefeed statement every time.

There is a use for scheduled/incremental exports: running a changefeed during off-hours to not disrupt SQL traffic. Some code needs to be added to actually make the scheduled run incremental (ie. maybe use CURSOR + end_time or transformatinos with protected timestamp management).