redpanda-data / redpanda

Redpanda is a streaming data platform for developers. Kafka API compatible. 10x faster. No ZooKeeper. No JVM!
https://redpanda.com
9.69k stars 591 forks source link

datalake/translation: prevent offset gaps and stuck translation #24247

Closed bharathv closed 2 days ago

bharathv commented 2 days ago

Normally gaps shouldn't happen with translation due to the enforcement of max_collectible_offset. However the following sequence of actions can create a gap.

iceberg enabled iceberg disabled <-- compaction --> iceberg enabled

This is an unfixable gap if compaction cleaned up an offset range adjacent to last translated offset. The fix just plugs the gap by adjusting the begin offset of the range that is to be committed with the coordinator. This is a rare case but can result in a stuck translation if it happens.

Backports Required

Release Notes