search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.1k stars 3.81k forks source link

cdc: revisit pts related cluster settings #129529

Open wenyihu6 opened 2 months ago

wenyihu6 commented 2 months ago

Is your feature request related to a problem? Please describe.

Previously, we always update PTS record if the last time we updated pts has passed by changefeed.protect_timestamp_interval and if a checkpoint happens. However, the lastProtectedTimestampUpdate is saved on frontiers, so this info is lost whenever chanegfeed restarts. This could lead to lots of PTS updates if changefeeds keep restarting.

To solve this problem, we introduced another cluster setting to allow PTS updates only if it is lagging behind high watermark by changefeed.protect_timestamp.lag in this PR https://github.com/cockroachdb/cockroach/pull/129442.

We now have two cluster settings to guard how frequent PTS updates happen. It makes things harder to rationalize. We should discuss and see if we should just keep one.

Jira issue: CRDB-41575

blathers-crl[bot] commented 2 months ago

cc @cockroachdb/cdc

wenyihu6 commented 2 months ago

In addition, we should also see how we can refactor this code for manageProtectedTimestamps.