KIP-405: 2023-08-18

[jeqo]

NOTE: not meant to be merged, but to create a working snapshot of the current work ongoing on KIP-405. We expect these fixes to land on the upstream trunk eventually.

Branch pulling latest changes related to Kafka Tiered Storage, KIP-405.

Includes the following PRs:

• KAFKA-14888: Added remote log segments retention functionality based on time and size
  • Upstream PR: https://github.com/apache/kafka/pull/13561
  • Changes squashed into https://github.com/aiven/kafka/tree/kips/KIP-405/feat/KAFKA-14888
• KAFKA-15181: Wait for RemoteLogMetadataCache to initialize after assigning partitions
  • Upstream PR: https://github.com/apache/kafka/pull/14127
  • Changes squashed into https://github.com/aiven/kafka/tree/kips/KIP-405/feat/KAFKA-15181
• KAFKA-15261: Do not block replica fetcher if RLMM is not initialized
  • Upstream PR: https://github.com/apache/kafka/pull/14112
  • Changes squashed into https://github.com/aiven/kafka/tree/kips/KIP-405/feat/KAFKA-15261
• KAFKA-15260: RLM Task should wait for RLMM to initialize
  • Upstream PR: https://github.com/apache/kafka/pull/14113
  • Changes squashed into https://github.com/aiven/kafka/tree/kips/KIP-405/feat/KAFKA-15260

Squashed changes are cherry-picked and small fixes are included on top of the upstream trunk.

A docker image can be built with the following tag: aivenoy/kafka:kips_KIP-405_2023-08-18

[jeqo]

Replaced by https://github.com/aiven/kafka/pull/38