create a standard set of benchmarks for manual compactions

[bananabrick]

Time taken by a manual compaction to de-invert an lsm is unknown. Some factors which may influence this time are:

1. File sizes
2. Number of keys per file
3. Number of files which have range delete blocks
4. Amount of file overlap
5. Overall size of L0
6. Number of sublevels in L0

It'll be nice to have a standard set of roachperf benchmark for a few variations of inverted lsms to figure out time taken to de-invert L0 for various L0 configurations. The results will allow us to make better recommendations during on-call emergencies.

Jira issue: PEBBLE-98

[nicktrav]

One thing we could consider is concocting something like a “bad LSM” state as a test / benchmark fixture (and storing somewhere, or building the generator into the pebble binary itself, like we do for cockroach workload). A few different types of “bad LSMs” - classic inversion (large L0 sublevel count / size); the newer single L0 sublevel / large size type - would be interesting.

These fixtures could then be used to build something like a consistent / repeatable nightly benchmark.

The results could also be surfaced on the Pebble nightly benchmarks page for tracking improvements over time.

[bananabrick]

Good idea! Although, this kind of thing might be better as a weekly benchmark because we'd still have to run each benchmark for each lsm shape a few times to get rid of the machine to machine variance. Maybe that's too expensive for a nightly bench.

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to Pebble!