Open mmaslankaprv opened 2 months ago
The problem that causes node 1
decommission to stop is related with stuck recovery of kafka/migration-test-workload/0
partition.
recovery_stm
requests a read:
TRACE 2024-06-19 13:11:34,923 [shard 1:raft] raft - [follower: {id: 6, revision: 170}] [group_id:64, {kafka/migration-test-workload/0}] - recovery_stm.cc:261 - Reading batches, starting from: 6973
It is stopped after more than 2 minutes of waiting:
TRACE 2024-06-19 13:13:52,671 [shard 1:raft] raft - [follower: {id: 6, revision: 170}] [group_id:64, {kafka/migration-test-workload/0}] - recovery_stm.cc:270 - Read no batches for recovery, stopping
The recovery continues after compaction finishes:
DEBUG 2024-06-19 13:13:52,671 [shard 1:lc ] storage-gc - disk_log_impl.cc:790 - [{kafka/migration-test-workload/0}] Final compacted segment {offset_tracker:{term:1, base_offset:7192, committed_offset:7447, dirty_offset:7447}, compacted_segment=1, finished_self_compaction=1, finished_windowed_compaction=1, generation=89, reader={/var/lib/redpanda/data/kafka/migration-test-workload/0_41/7192-1-v1.log, (92511 bytes)}, writer=nullptr, cache=nullptr, compaction_index:nullopt, closed=0, tombstone=0, index={file:/var/lib/redpanda/data/kafka/migration-test-workload/0_41/7192-1-v1.base_index, offsets:0, index:{header_bitflags:0, base_offset:0, max_offset:7447, base_timestamp:{timestamp: 1718802677049}, max_timestamp:{timestamp: 1718802677196}, batch_timestamps_are_monotonic:1, with_offset:true, non_data_timestamps:0, broker_timestamp:{{timestamp: 1718802677191}}, num_compactible_records_appended:{22}, index(3,3,3)}, step:32768, needs_persistence:0}}
ci-ignore because https://redpandadata.slack.com/archives/C06MCRY75U7/p1718996592057029
https://buildkite.com/redpanda/redpanda/builds/50454#01903098-f01c-43b5-9802-57743666b009
JIRA Link: CORE-4258