search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.88k stars 3.77k forks source link

kvserver: re-evaluate raft log truncation heuristics #75802

Open tbg opened 2 years ago

tbg commented 2 years ago

Is your feature request related to a problem? Please describe.

The raft log truncation heuristics date back to a world in which we were very aggressive about truncations (since the log needed to be included in snapshots and could make them very large) while simultaneously being very defensive (if a snapshot is in flight, we will never cut it off from the log).

We just saw (internal link) that these heuristics could be problematic, and that some of the thresholds were not adjusted when we changed our default range size from 64mb to 512mb.

We should revisit all of these heuristics and, through thought and experimentation, adjust them with the goal of avoiding pathological build-up of raft log as well as "unnecessary" rejection of snapshots

Additional context

https://github.com/cockroachdb/cockroach/issues/36262 also proposes allowing followers to truncate "locally" (i.e. without being prompted to truncated by the leader), in which case they would use different (simpler) heuristics than the leader.

Jira issue: CRDB-12843

Epic CRDB-39898

erikgrinaker commented 2 years ago

Additionally, #75793 adds a raftLogTruncationClearRangeThreshold constant that controls the number of entries beyond which log truncation switches from dropping Pebble point tombstones to range tombstones. This is set very conservatively (100000), and is unlikely to trigger even with lagging followers and hitting RaftLogTruncationThreshold (16 MB), due to concerns that dropping too many range tombstones may be detrimental to Pebble.

A setting of e.g. 1000 appears more reasonable here, but this needs to be tuned in concert with e.g. RaftLogTruncationThreshold and other parameters, and benchmarked to make sure the range tombstones are an improvement rather than a detriment in most or all realistic circumstances. Also related to cockroachdb/pebble#1295.

github-actions[bot] commented 9 months ago

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to CockroachDB!