Compaction improvements

dotnwat commented 11 months ago

Problem

Redpanda currently has some limitations in its compaction:

The set of keys considered during compaction are constrained to both (a) a single term and (b) single segment (or combined adjacent segments). This limits the scope of keys, and in turn, the compaction ratio that we can achieve.
Configuration batches are not compacted
Tombstones are not removed

Solution

The scope of keys considered will not be constrained to a single term, nor a single segment. This will improve compaction ratio.
We will not merge segments from different terms, which means we'll have at least one segment per term. In order to detail with many segments due to many terms, or many small segments following compaction, we'll continue to use adjacent segment merging/compaction technique.
Identify the point at which configuration batches and tombstones are safe to remove.

Impact

Applications that make intensive use of compaction will no longer risk experiencing unexpectedly high consumption of disk space and system resources on redpanda nodes.

### Tasks
- [ ] https://github.com/redpanda-data/redpanda/issues/14363
- [ ] https://github.com/redpanda-data/redpanda/issues/14365
- [ ] https://github.com/redpanda-data/redpanda/issues/14367
- [ ] https://github.com/redpanda-data/redpanda/issues/14364
- [ ] https://github.com/redpanda-data/redpanda/issues/14366
- [ ] https://github.com/redpanda-data/redpanda/issues/14858
- [x] Fix Raft truncation racing with compactions https://redpandadata.slack.com/archives/C044RD18NMV/p1700080049179969?thread_ts=1699990143.475119&cid=C044RD18NMV
- [ ] https://github.com/redpanda-data/redpanda/pull/14994
- [ ] https://github.com/redpanda-data/redpanda/issues/15143
- [ ] https://github.com/redpanda-data/redpanda/issues/15043
- [x] Memory, oh my! https://redpandadata.slack.com/archives/C04B8659KGR/p1701457121387139
- [ ] https://github.com/redpanda-data/redpanda/issues/15347
- [ ] https://github.com/redpanda-data/redpanda/pull/15415
- [ ] https://github.com/redpanda-data/redpanda/pull/15416

### Immediate follow-up / nice to have
- [ ] https://github.com/redpanda-data/redpanda/issues/14372
- [ ] https://github.com/redpanda-data/redpanda/issues/14410
- [ ] https://github.com/redpanda-data/redpanda/issues/14947

### Longer term follow-up / nice to have
- [ ] https://github.com/redpanda-data/redpanda/issues/14369
- [ ] https://github.com/redpanda-data/redpanda/issues/14370

JIRA Link: CORE-1487

jnoynaert commented 11 months ago

FYI, as a consideration for this one: the equivalent of Kafka's delete.retention.ms (to guarantee tombstone lifetimes) would be required for us to upgrade to any version that removes tombstones during compaction.

Feediver1 commented 11 months ago

According to PM, this requires a doc update--adding the doc-needed label...

redpanda-data / redpanda