andrewbaptist
commented
2 weeks ago One consideration for the test is to do the following:
1) Set up a cluster with 12 nodes.
2) Run a KV workload with large values (10K) to fill the system up sufficiently that there are multiple levels of the LSM
3) Manually run "full compaction" of the LSM (which makes the LSM very healthy)
4) Stop the cluster and start with a lower range lease duration: the range lease duration turned down COCKROACH_RANGE_LEASE_DURATION to something low (try 0.5s). And let the cluster sit for a while (4-6 hours).
5) Restart the KV workload at a low enough rate to not max out the CPU.
6) Wait until enough point tombstones have accumulated and try and move the liveness lease (or do a clean shutdown of the liveness leaseholder node).
I'm not sure if this will reproduce it, but something like this likely will.
As described in https://github.com/cockroachdb/pebble/issues/918, certain workloads are prone to buildup of point tombstones at the storage layer, which can significantly degrade read latency. This was addressed in https://github.com/cockroachdb/pebble/pull/3790, but we should have a roachtest to demonstrate the performance improvement at the CockroachDB level.
Workloads involving a queue are especially susceptible to this issue. The section titled "CockroachDB Workload" in the comment here describes a queue-based workload where we saw read performance slowly worsen over time as tombstones build up. This could be adapted directly into a roachtest, but it may also be worth investigating other workload types since average latency still only degrades from 1ms -> 7ms in that case before the tombstones are compacted away by previously existing heuristics. We have seen cases where reads can take multiple seconds, e.g. in the liveness range (https://github.com/cockroachdb/cockroach/issues/128968).
Jira issue: CRDB-41649