storage: adjacent segment merging for non-compacted topics

[jcsp]

What

Even if a topic doesn't want compaction of its data, we should still consider merging adjacent segments, including dropping configuration batches where possible.

Why

This is especially relevant in unhealthy clusters, where repeated raft elections may lead to large numbers of tiny segments, with associated costs in memory & file handles, with extreme cases such as crash loops leading to outages.

How

We should do adjacent segment compaction when:

• There are adjacent segments with only one batch each: this is the pathological case of tiny segments resulting from raft elections without any user writes between them.
• The average segment size is below some threshold (e.g. say, 10% of the configured segment.bytes for the partition). This for for small but not pathologically small segments, which could result perhaps from partitions with very low data rates, which end up with small segments from periodic leadership change events like routine restarts.

The main thing to be careful of is raft terms: when we merge segments we may discard the knowledge of what term particular batches were in: this is only safe if we're sure all raft peers are up to date and will not need to replay from further back. This is a hand-waving explanation: a more rigorous think-through of the raft consequences will be needed in https://github.com/redpanda-data/redpanda/issues/6432 when implementing cross-term merges &

As well as doing adjacent segment merging as a background housekeeping task, we should sometimes do it at startup: if a topic has several single-batch segments at the tip of the log on startup, we should merge them before starting raft if it is safe to do so, so that if we are in some kind of restart loop, we do not create new files indefinitely.

Related tickets:

• Dependency: https://github.com/redpanda-data/redpanda/issues/6432 (includes doing cross-term merging of segments when safe, and adding code for stripping out configuration batches that are no longer needed)
• https://github.com/redpanda-data/redpanda/issues/6433 (the equivalent functionality for cloud storage, this ticket is about local storage)
• https://github.com/redpanda-data/redpanda/issues/6569 (for the safety net that prevents indefinite retry loops: combined with this ticket, we will have two layers of protection against FD exhaustion)

JIRA Link: CORE-1039

[jcsp]

@mmaslankaprv makes the excellent point that once we have the ability to put data from different terms in the same segment, we might use that to avoid opening new segments on restart at all, and instead pick up and continue appending to an existing segment.