db: improve visibility into snapshots

[jbowens]

Open snapshots can impact performance. During compactions, they increase write amplification by requiring us to write keys that we otherwise could elide. During iteration, iterators must sift through obsolete keys preserved by snapshots. The compensated size compaction heuristic doesn't take snapshots into account (see #872), which can exacerbate the above issues by prioritizing compactions that must write obsolete keys.

We have some ideas (eg, #847) in lessening the impact of snapshots. It's hard to prioritize any work here without understanding how much of an impact snapshots have on real workloads. Some ideas for improvising visibility here:

• Add a obsolete.keys.count property to sstables, counting the total number of keys within an sstable that could have been elided (from range deletions, point deletions, overwriting, etc) if there were no open snapshots.
• Add a obsolete.keys.size property to sstables, summing the raw key size and raw value size of the above.
• Add a manifest.Annotator that sums the above sstable properties over an entire level and expose on pebble.Metrics.
• Expose the count of open snapshots and the earliest sequence number (or real time, but we'd need to start collecting it) of an open snapshot on pebble.Metrics.

Jira issue: PEBBLE-208

Epic CRDB-40360

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to Pebble!

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to Pebble!