sstable: explore index size optimizations

[petermattis]

RocksDB has added version 3 and version 4 sstable formats:

• Version 3 changes index block keys to not include the sequence number (which is always 0?)
• Version 4 changes index block values (block handles) from being encoded as <offset, size> to a delta encoding that usually only stores <size>.

There is no particular urgency to supporting these formats, but we should investigate whether these format changes move the needle on any benchmarks.

Jira issue: PEBBLE-185

[petermattis]

There is no particular urgency to supporting these formats, but we should investigate whether these format changes move the needle on any benchmarks.

@ajkr do you have an inside knowledge on this?

[ajkr]

I remember format_version=4 saved noticeable space on indexes. But it looks like it depends on setting index_block_restart_interval > 1, which we currently don't do. So it shouldn't be the first thing we try if we ever want to optimize index size.

[petermattis]

So it shouldn't be the first thing we try if we ever want to optimize index size.

Something I've been meaning to ask you about: we saw a situation once where keys larger than a data block were written to a table. The result was that every data block had 1 entry, and the index block was essentially the same size as all of the data blocks combined. I think we could avoid this situation if there was a bit more complexity in the sstable block boundary heuristics. Specifically, we shouldn't allow a block to contain only a single entry if the key for the entry is larger than the target block size.

On the other hand, perhaps it isn't worth futzing with the sstable block boundary heuristics, and instead put effort into blob (wisc-key) storage of large entries.

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to Pebble!

[github-actions[bot]]

We have marked this issue as stale because it has been inactive for 18 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to Pebble!