search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
7.37k stars 3.11k forks source link

ListOffsets loop of failed requests on leader epoch change until timeout happens #4620

Closed emasab closed 2 months ago

emasab commented 4 months ago

Description

ListOffsets requests done for partitions with no committed offsets can be retried indefinitely if that partition leader epoch has changed, because the buffer is retried without recreating it with the new CurrentLeaderEpoch received from the Metadata refresh call.

How to reproduce

Start consuming partitions that have no committed offset, or seek to the latest offset. A partition leader change should happen that changes the current leader epoch to a value higher than the cached one. The ListOffsets request give a FENCED_LEADER_EPOCH and then it refreshes Metadata, but starts retrying the buffer with the same CurrentLeaderEpoch, leading to a loop of failed requests.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

davidblewett commented 3 months ago

@emasab we have several corroborating reports of this bug with v2.3.0. Is the change in https://github.com/confluentinc/librdkafka/pull/4616 reasonable?

emasab commented 3 months ago

@davidblewett no, the two issues seem unrelated. We're fixing this one though

emasab commented 2 months ago

While for the other one we're evaluating a fix, this one isn't occurring, as a metadata refresh is requested and the ListOffsets request is migrated to the new leader after a leader epoch update and a new ListOffset request is made.

Some extracted logs when it's returning "Not leader for partition"

%7|1712742491.308|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-query -> offset-wait
%7|1712742491.308|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: ListOffsetsRequest (v7, opv 3) for 1 topic(s) and 1 partition(s)
%7|1712742491.309|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 17)
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received ListOffsetsRequestV7 from 127.0.0.1:39396
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset -1 (leader epoch 2) for END: Broker: Not leader for partition
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending ListOffsetsResponseV7 to 127.0.0.1:39396
%7|1712742491.309|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Received ListOffsetsResponse (v7, 41 bytes, CorrId 17, rtt 0.19ms)
%7|1712742491.309|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: Offset reply for topic test [0] (v3 vs v3)
%7|1712742491.309|REQERR|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: ListOffsetsRequest failed: Broker: Not leader for partition: explicit actions Refresh
%7|1712742491.309|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: OffsetRequest failed: Broker: Not leader for partition (Refresh)
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Requesting metadata for 1/1 topics: ListOffsetsRequest failed: Broker: Not leader for partition
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: Request metadata for 1 topic(s): ListOffsetsRequest failed: Broker: Not leader for partition
%7|1712742491.309|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: Offset reply error for topic test [0] (v3, Broker: Not leader for partition): Refresh
%7|1712742491.309|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: failed to query logical offset END: Broker: Not leader for partition: (re)starting offset query timer for offset END (leader epoch -1)
%7|1712742491.309|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-wait -> offset-query
%7|1712742491.309|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Sent MetadataRequest (v9, 55 bytes @ 0, CorrId 18)
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received MetadataRequestV9 from 127.0.0.1:39396
%7|1712742491.309|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending MetadataResponseV9 to 127.0.0.1:39396
%7|1712742491.309|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Received MetadataResponse (v9, 283 bytes, CorrId 18, rtt 0.20ms)
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: ===== Received metadata (for 1 requested topics): ListOffsetsRequest failed: Broker: Not leader for partition =====
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: ClusterId: mockCluster1fd2f8000378, ControllerId: 0
%7|1712742491.309|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: 3 brokers, 1 topics
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2:   Broker #0/3: 127.0.0.1:46459 NodeId 1
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2:   Broker #1/3: 127.0.0.1:36427 NodeId 2
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2:   Broker #2/3: 127.0.0.1:34863 NodeId 3
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2:   Topic test with 4 partitions
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 0 Leader 3 Epoch 2
%7|1712742491.310|BROKER|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: leader 1 epoch 1 -> leader 3 epoch 2
%7|1712742491.310|TOPICUPD|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Topic test [0]: migrating from broker 2 to 3 (leader is 3): leader updated
%7|1712742491.310|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: delegate to broker 127.0.0.1:34863/3 (rktp 0x7f4bd0001850, term 0, ref 6)
%7|1712742491.310|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: no longer delegated to broker 127.0.0.1:36427/2
%7|1712742491.310|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: delegating to broker 127.0.0.1:34863/3 for partition with 0 messages (0 bytes) queued
%7|1712742491.310|BRKMIGR|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Migrating topic test [0] 0x7f4bd0001850 from 127.0.0.1:36427/2 to 127.0.0.1:34863/3 (sending PARTITION_LEAVE to 127.0.0.1:36427/2)
%7|1712742491.310|VALIDATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: skipping offset validation in fetch state offset-query
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 1 Leader 1 Epoch 0
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 2 Leader 2 Epoch 0
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 3 Leader 2 Epoch 0
%7|1712742491.310|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:36427/2: 1/1 requested topic(s) seen in metadata
%7|1712742491.310|TOPBRK|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: 127.0.0.1:36427/2: Topic test [0]: leaving broker (0 messages in xmitq, next broker 127.0.0.1:34863/3, rktp 0x7f4bd0001850)
%7|1712742491.310|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:36427/bootstrap]: Broadcasting state change
%7|1712742491.310|TOPBRK|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Topic test [0]: joining broker (rktp 0x7f4bd0001850, 0 message(s) queued)
%7|1712742491.310|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
[<MAIN>                      /  6.001s] 1 test(s) running: 0104_fetch_from_follower_mock
%7|1712742491.810|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Topic test [0]: timed offset query for offset END (leader epoch -1) in state offset-query
%7|1712742491.810|OFFREQ|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:34863/3: Partition test [0]: querying for logical offset END (opv 3)
%7|1712742491.810|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-query -> offset-wait
%7|1712742491.810|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker changed state INIT -> TRY_CONNECT
%7|1712742491.810|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
%7|1712742491.810|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: broker in state TRY_CONNECT connecting
%7|1712742491.810|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker changed state TRY_CONNECT -> CONNECT
%7|1712742491.810|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
%7|1712742491.810|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Connecting to ipv4#127.0.0.1:34863 (plaintext) with socket 29
%7|1712742491.810|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Connected to ipv4#127.0.0.1:34863
%7|1712742491.810|CONNECTED|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Connected (#1)
%7|1712742491.810|FEATURE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1712742491.810|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1712742491.810|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
%7|1712742491.810|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Sent ApiVersionRequest (v3, 79 bytes @ 0, CorrId 1)
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: New connection from 127.0.0.1:56786
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ApiVersionRequestV3 from 127.0.0.1:56786
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ApiVersionResponseV3 to 127.0.0.1:56786
%7|1712742491.810|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Received ApiVersionResponse (v3, 13 bytes, CorrId 1, rtt 0.11ms)
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
%7|1712742491.810|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Sent ApiVersionRequest (v0, 43 bytes @ 0, CorrId 2)
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ApiVersionRequestV0 from 127.0.0.1:56786
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ApiVersionResponseV0 to 127.0.0.1:56786
%7|1712742491.810|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Received ApiVersionResponse (v0, 114 bytes, CorrId 2, rtt 0.09ms)
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker API support:
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey Produce (0) Versions 0..7
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey Fetch (1) Versions 0..11
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey ListOffsets (2) Versions 0..7
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey Metadata (3) Versions 0..9
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey OffsetCommit (8) Versions 0..8
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey OffsetFetch (9) Versions 0..6
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey FindCoordinator (10) Versions 0..3
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey JoinGroup (11) Versions 0..6
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey Heartbeat (12) Versions 0..5
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey LeaveGroup (13) Versions 0..4
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey SyncGroup (14) Versions 0..4
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey ApiVersion (18) Versions 0..2
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey InitProducerId (22) Versions 0..4
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey OffsetForLeaderEpoch (23) Versions 2..2
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey AddPartitionsToTxn (24) Versions 0..1
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey AddOffsetsToTxn (25) Versions 0..1
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey EndTxn (26) Versions 0..1
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:   ApiKey TxnOffsetCommit (28) Versions 0..3
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature MsgVer1: Produce (2..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature MsgVer1: Fetch (2..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature MsgVer1
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature MsgVer2: Produce (3..3) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature MsgVer2: Fetch (4..4) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature MsgVer2
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ApiVersion: ApiVersion (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature ApiVersion
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerGroupCoordinator: FindCoordinator (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature BrokerGroupCoordinator
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: FindCoordinator (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature BrokerBalancedConsumer
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ThrottleTime: Produce (1..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ThrottleTime: Fetch (1..2) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature ThrottleTime
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature Sasl: JoinGroup (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature Sasl
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature SaslHandshake: SaslHandshake (0..0) NOT supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Disabling feature SaslHandshake
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature LZ4: FindCoordinator (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature LZ4
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature OffsetTime: ListOffsets (1..1) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature OffsetTime
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature IdempotentProducer: InitProducerId (0..0) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature IdempotentProducer
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ZSTD: Produce (7..7) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature ZSTD: Fetch (10..10) supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Enabling feature ZSTD
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature SaslAuthReq: SaslHandshake (1..1) NOT supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3:  Feature SaslAuthReq: SaslAuthenticate (0..1) NOT supported by broker
%7|1712742491.810|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Disabling feature SaslAuthReq
%7|1712742491.810|FEATURE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD
%7|1712742491.810|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Broker changed state APIVERSION_QUERY -> UP
%7|1712742491.810|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Broadcasting state change
%7|1712742491.810|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: Skipping metadata refresh of 1 topic(s): connected: already being requested
%7|1712742491.810|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: ListOffsetsRequest (v7, opv 3) for 1 topic(s) and 1 partition(s)
%7|1712742491.810|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:34863/bootstrap]: 127.0.0.1:34863/3: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 3)
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ListOffsetsRequestV7 from 127.0.0.1:56786
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset 1000 (leader epoch 2) for END: Success
%7|1712742491.810|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ListOffsetsResponseV7 to 127.0.0.1:56786

Even when we allow to list offsets on the follower with ReplicaId -2 and the error becomes Leader epoch is older than broker epoch the retrying buffer is discarded because outdated and a new request is made to the new leader:

%7|1712743038.657|RETRY|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Moved 1 retry buffer(s) to output queue
%7|1712743038.657|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 20)
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received ListOffsetsRequestV7 from 127.0.0.1:33182
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset -1 (leader epoch 2) for END: Broker: Leader epoch is older than broker epoch
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending ListOffsetsResponseV7 to 127.0.0.1:33182
%7|1712743038.657|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Received ListOffsetsResponse (v7, 41 bytes, CorrId 20, rtt 0.28ms)
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply for topic test [0] (v3 vs v3)
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply error for topic test [0] (v3, Local: Outdated):

Full logs after allowing to ListOffsets on a follower

%7|1712743038.557|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-query -> offset-wait
%7|1712743038.557|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: ListOffsetsRequest (v7, opv 3) for 1 topic(s) and 1 partition(s)
%7|1712743038.557|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 18)
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received ListOffsetsRequestV7 from 127.0.0.1:33182
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset -1 (leader epoch 2) for END: Broker: Leader epoch is older than broker epoch
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending ListOffsetsResponseV7 to 127.0.0.1:33182
%7|1712743038.557|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Received ListOffsetsResponse (v7, 41 bytes, CorrId 18, rtt 0.17ms)
%7|1712743038.557|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply for topic test [0] (v3 vs v3)
%7|1712743038.557|REQERR|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: ListOffsetsRequest failed: Broker: Leader epoch is older than broker epoch: explicit actions Refresh,Retry
%7|1712743038.557|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: OffsetRequest failed: Broker: Leader epoch is older than broker epoch (Refresh,Retry)
%7|1712743038.557|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Requesting metadata for 1/1 topics: ListOffsetsRequest failed: Broker: Leader epoch is older than broker epoch
%7|1712743038.557|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Request metadata for 1 topic(s): ListOffsetsRequest failed: Broker: Leader epoch is older than broker epoch
%7|1712743038.557|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply error for topic test [0] (v3, Local: Operation in progress): Refresh,Retry
%7|1712743038.557|RETRY|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Retrying ListOffsetsRequest (v7, 75 bytes, retry 1/2, prev CorrId 18) in 100ms
%7|1712743038.557|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Sent MetadataRequest (v9, 55 bytes @ 0, CorrId 19)
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received MetadataRequestV9 from 127.0.0.1:33182
%7|1712743038.557|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending MetadataResponseV9 to 127.0.0.1:33182
%7|1712743038.558|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Received MetadataResponse (v9, 283 bytes, CorrId 19, rtt 0.11ms)
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: ===== Received metadata (for 1 requested topics): ListOffsetsRequest failed: Broker: Leader epoch is older than broker epoch =====
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: ClusterId: mockCluster1fd4e7000378, ControllerId: 0
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: 3 brokers, 1 topics
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2:   Broker #0/3: 127.0.0.1:42445 NodeId 1
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2:   Broker #1/3: 127.0.0.1:40441 NodeId 2
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2:   Broker #2/3: 127.0.0.1:40643 NodeId 3
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2:   Topic test with 4 partitions
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 0 Leader 3 Epoch 2
%7|1712743038.558|BROKER|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: leader 1 epoch 1 -> leader 3 epoch 2
%7|1712743038.558|TOPICUPD|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Topic test [0]: migrating from broker 2 to 3 (leader is 3): leader updated
%7|1712743038.558|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: delegate to broker 127.0.0.1:40643/3 (rktp 0x7f5390001850, term 0, ref 7)
%7|1712743038.558|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: no longer delegated to broker 127.0.0.1:40441/2
%7|1712743038.558|BRKDELGT|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: delegating to broker 127.0.0.1:40643/3 for partition with 0 messages (0 bytes) queued
%7|1712743038.558|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: migrating to new broker: (re)starting offset query timer for offset END (leader epoch -1)
%7|1712743038.558|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-wait -> offset-query
%7|1712743038.558|BRKMIGR|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Migrating topic test [0] 0x7f5390001850 from 127.0.0.1:40441/2 to 127.0.0.1:40643/3 (sending PARTITION_LEAVE to 127.0.0.1:40441/2)
%7|1712743038.558|VALIDATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: skipping offset validation in fetch state offset-query
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 1 Leader 1 Epoch 0
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 2 Leader 1 Epoch 0
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]:   Topic test partition 3 Leader 2 Epoch 0
%7|1712743038.558|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: 1/1 requested topic(s) seen in metadata
%7|1712743038.558|TOPBRK|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Topic test [0]: leaving broker (0 messages in xmitq, next broker 127.0.0.1:40643/3, rktp 0x7f5390001850)
%7|1712743038.558|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: Broadcasting state change
%7|1712743038.558|TOPBRK|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Topic test [0]: joining broker (rktp 0x7f5390001850, 0 message(s) queued)
%7|1712743038.558|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743038.657|RETRY|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Moved 1 retry buffer(s) to output queue
%7|1712743038.657|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 20)
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Received ListOffsetsRequestV7 from 127.0.0.1:33182
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset -1 (leader epoch 2) for END: Broker: Leader epoch is older than broker epoch
%7|1712743038.657|MOCK|MOCK#producer-1| [thrd:mock]: Broker 2: Sending ListOffsetsResponseV7 to 127.0.0.1:33182
%7|1712743038.657|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40441/bootstrap]: 127.0.0.1:40441/2: Received ListOffsetsResponse (v7, 41 bytes, CorrId 20, rtt 0.28ms)
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply for topic test [0] (v3 vs v3)
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40441/2: Offset reply error for topic test [0] (v3, Local: Outdated): 
%7|1712743038.657|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: test [0]: outdated offset response: offset query timer already scheduled for offset END (leader epoch -1)
[<MAIN>                      /  8.001s] 1 test(s) running: 0104_fetch_from_follower_mock
%7|1712743039.058|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Topic test [0]: timed offset query for offset END (leader epoch -1) in state offset-query
%7|1712743039.058|OFFREQ|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: 127.0.0.1:40643/3: Partition test [0]: querying for logical offset END (opv 3)
%7|1712743039.058|PARTSTATE|0104_fetch_from_follower_mock#consumer-3| [thrd:main]: Partition test [0] changed fetch state offset-query -> offset-wait
%7|1712743039.059|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker changed state INIT -> TRY_CONNECT
%7|1712743039.059|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743039.059|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: broker in state TRY_CONNECT connecting
%7|1712743039.059|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker changed state TRY_CONNECT -> CONNECT
%7|1712743039.059|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743039.059|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Connecting to ipv4#127.0.0.1:40643 (plaintext) with socket 29
%7|1712743039.059|CONNECT|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Connected to ipv4#127.0.0.1:40643
%7|1712743039.059|CONNECTED|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Connected (#1)
%7|1712743039.059|FEATURE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: New connection from 127.0.0.1:49350
%7|1712743039.059|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1712743039.059|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743039.059|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Sent ApiVersionRequest (v3, 79 bytes @ 0, CorrId 1)
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ApiVersionRequestV3 from 127.0.0.1:49350
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ApiVersionResponseV3 to 127.0.0.1:49350
%7|1712743039.059|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Received ApiVersionResponse (v3, 13 bytes, CorrId 1, rtt 0.17ms)
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: ApiVersionRequest v3 failed due to UNSUPPORTED_VERSION: retrying with v0
%7|1712743039.059|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Sent ApiVersionRequest (v0, 43 bytes @ 0, CorrId 2)
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ApiVersionRequestV0 from 127.0.0.1:49350
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Sending ApiVersionResponseV0 to 127.0.0.1:49350
%7|1712743039.059|RECV|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Received ApiVersionResponse (v0, 114 bytes, CorrId 2, rtt 0.16ms)
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker API support:
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey Produce (0) Versions 0..7
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey Fetch (1) Versions 0..11
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey ListOffsets (2) Versions 0..7
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey Metadata (3) Versions 0..9
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey OffsetCommit (8) Versions 0..8
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey OffsetFetch (9) Versions 0..6
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey FindCoordinator (10) Versions 0..3
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey JoinGroup (11) Versions 0..6
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey Heartbeat (12) Versions 0..5
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey LeaveGroup (13) Versions 0..4
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey SyncGroup (14) Versions 0..4
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey ApiVersion (18) Versions 0..2
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey InitProducerId (22) Versions 0..4
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey OffsetForLeaderEpoch (23) Versions 2..2
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey AddPartitionsToTxn (24) Versions 0..1
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey AddOffsetsToTxn (25) Versions 0..1
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey EndTxn (26) Versions 0..1
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:   ApiKey TxnOffsetCommit (28) Versions 0..3
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature MsgVer1: Produce (2..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature MsgVer1: Fetch (2..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature MsgVer1
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature MsgVer2: Produce (3..3) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature MsgVer2: Fetch (4..4) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature MsgVer2
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ApiVersion: ApiVersion (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature ApiVersion
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerGroupCoordinator: FindCoordinator (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature BrokerGroupCoordinator
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: FindCoordinator (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature BrokerBalancedConsumer
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ThrottleTime: Produce (1..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ThrottleTime: Fetch (1..2) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature ThrottleTime
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature Sasl: JoinGroup (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature Sasl
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature SaslHandshake: SaslHandshake (0..0) NOT supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Disabling feature SaslHandshake
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature LZ4: FindCoordinator (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature LZ4
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature OffsetTime: ListOffsets (1..1) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature OffsetTime
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature IdempotentProducer: InitProducerId (0..0) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature IdempotentProducer
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ZSTD: Produce (7..7) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature ZSTD: Fetch (10..10) supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Enabling feature ZSTD
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature SaslAuthReq: SaslHandshake (1..1) NOT supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3:  Feature SaslAuthReq: SaslAuthenticate (0..1) NOT supported by broker
%7|1712743039.059|APIVERSION|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Disabling feature SaslAuthReq
%7|1712743039.059|FEATURE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD
%7|1712743039.059|STATE|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Broker changed state APIVERSION_QUERY -> UP
%7|1712743039.059|BROADCAST|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Broadcasting state change
%7|1712743039.059|METADATA|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: Skipping metadata refresh of 1 topic(s): connected: already being requested
%7|1712743039.059|OFFSET|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: ListOffsetsRequest (v7, opv 3) for 1 topic(s) and 1 partition(s)
%7|1712743039.059|SEND|0104_fetch_from_follower_mock#consumer-3| [thrd:127.0.0.1:40643/bootstrap]: 127.0.0.1:40643/3: Sent ListOffsetsRequest (v7, 75 bytes @ 0, CorrId 3)
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Broker 3: Received ListOffsetsRequestV7 from 127.0.0.1:49350
%7|1712743039.059|MOCK|MOCK#producer-1| [thrd:mock]: Topic test [0] returning offset 1000 (leader epoch 2) for END: Success