Large values: overflow blocks

[danthegoodman1]

If we obtain a value that would overflow the block, instead of expanding the block size, it is implied that the value continues in the next block (because we have the size of the next key and value).

This is safe because we always read the data block forward. Then at the end of that value, the next KV starts.

The segment reader will have to be adjusted to be able to buffer values it has partially.

Whether this performance matters for reasonably max size values (e.g. 16KB) probably doesn't matter much, but if something like a 4GB max is permitted then it might make a substantial difference (but this is probably quite bad practice).

[danthegoodman1]

If a key would result in overflowing the block, then the current block is ended and a new one started. If a value would result in an overflowing block, then it is partially written to the current block, and continued as the first bytes of the next block.

This means that the block index may not have contiguous entries, as these "overflow blocks" wouldn't appear in the index.

[danthegoodman1]

This is similar to how SQLite handles large values that exceed a single page: It breaks them up across multiple pages.