storage: roachperf benchmark for cluster write amplification

[nicktrav]

Is your feature request related to a problem? Please describe.

We currently have a write throughput benchmark at the Pebble layer that is computed at nightly as part of the Pebble nightly benchmark roachtests. The output of these benchmarks is tracked here. This has proved valuable for understanding Pebble write performance and identifying bottlenecks (i.e. compaction concurrency / scheduling).

It would be beneficial to have a similar benchmark at the Cockroach level (i.e. above Pebble, and in a clustered setting) that allows us to gauge the impact of various potential improvements to write-amplification (virtual ssts, separated raft log, value blocks, a wisckey-style value log, etc.).

Describe the solution you'd like

Implement a roachtest (likely making use of #81516, once productionized) from which we could derive a metric indicating the write amplification for a given trickle workload (i.e. how does a 1MB/s "goodput" translate to bytes written (and re-written as part of compactions) in a cluster). Something like:

"node_exporter reported bytes written after 2h of 1mb/s kv0 + 1h cooldown period"

Describe alternatives you've considered

Continue relying on just the Pebble benchmarks. This doesn't provide any insight into Cockroach-level limitations / bottlenecks / improvements.

Additional context

Some additional threads (internal) with relevant commentary:

• round one - the original report
• round two
• round three - after fixing the EBS issue

Jira issue: CRDB-16889

[jbowens]

I wish we could collect and display a w-amp metric on existing roachtests. This would also help tell the story of how a w-amp (or r-amp) improvement leads to increased throughput. roachperf doesn't yet support this: cockroachdb/roachperf#72