cdc: handle pts contention better

HonoreDB commented 1 year ago

When running many changefeeds separately, the protected timestamp system can get overwhelmed, and we don't seem to have many knobs to tune how often we make calls. We should do some or all of the following:

[ ] Add/document a cluster setting to reduce the number of calls a changefeed makes to advance protected timestamps.
[ ] Don't restart the feed when we fail to advance a protected timestamp
[ ] Retry with backoff when trying to create a protected timestamp
[ ] Automatically throttle the rate at which we advance protected timestamps in response to backpressure from the pts.

Jira issue: CRDB-25344

Epic CRDB-28882

blathers-crl[bot] commented 1 year ago

cc @cockroachdb/cdc

miretskiy commented 1 year ago

What kind of contention are we talking about? Do we have observations that show this to be the problem?

Changefeed does have the knobs to slow down creation -- we won't write anything more than once every 30 seconds. Many changefeeds -- well, that's another question -- I suspect all writes go to the same range which may experience load -- but ... presumably it should split at some point?

amruss commented 1 year ago

Support issue: https://github.com/cockroachlabs/support/issues/2119

cockroachdb / cockroach

cdc: handle pts contention better #98585