cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.1k stars 3.8k forks source link

rangefeed: client-controlled closed timestamp interval #114134

Open erikgrinaker opened 11 months ago

erikgrinaker commented 11 months ago

In large clusters, closed timestamp processing accounts for a large fraction of changefeed resource usage. Often, the changefeed will run with a large checkpoint interval (default min_checkpoint_frequency = 30s), but the default rangefeed closed timestamp interval is much smaller (kv.rangefeed.closed_timestamp_refresh_interval = 3s), which can produce unnecessary overhead.

It would be convenient if the rangefeed client could request a specific refresh interval, to avoid sending these more often than needed.

This would also be useful for system rangefeeds, that are low-volume but may also want low latency (see e.g. span config updates which are only applied after a checkpoint).

Jira issue: CRDB-33364

Epic CRDB-39898

blathers-crl[bot] commented 11 months ago

cc @cockroachdb/replication