ElectricVampire
commented
1 year ago Logs :
%7|1692263746.477|SASL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Initializing SASL client: service name kafka, hostname b-3.test.kafka.us-east-1.amazonaws.com, m
echanisms SCRAM-SHA-512, provider SCRAM (builtin)
%7|1692263746.477|SASLSCRAM|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: SASL SCRAM client in state client-first-message
%7|1692263746.477|SASL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Send SASL Kafka frame to broker (60 bytes)
%7|1692263746.477|SEND|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Sent SaslAuthenticateRequest (v0, 85 bytes @ 0, CorrId 111189)
%7|1692263746.478|RECV|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Received SaslAuthenticateResponse (v0, 113 bytes, CorrId 111189, rtt 1.11ms)
%7|1692263746.478|SASL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Received SASL frame from broker (105 bytes)
%7|1692263746.478|SASLSCRAM|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: SASL SCRAM client in state server-first-message
%7|1692263746.487|SASL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Send SASL Kafka frame to broker (189 bytes)
%7|1692263746.487|SEND|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Sent SaslAuthenticateRequest (v0, 214 bytes @ 0, CorrId 111190)
%7|1692263746.489|RECV|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Received SaslAuthenticateResponse (v0, 112 bytes, CorrId 111190, rtt 1.11ms)
%7|1692263746.489|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCR
AM-SHA-512 (after 11ms in state AUTH_REQ) (_AUTHENTICATION): identical to last error: error log suppressed
%7|1692263746.489|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Broker changed state AUTH_REQ -> DOWN
%7|1692263746.489|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263746.489|BUFQ|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Purging bufq with 0 buffers
%7|1692263746.489|BUFQ|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Purging bufq with 0 buffers
%7|1692263746.489|BUFQ|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Updating 0 buffers on connection reset
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: Requesting metadata for 1/1 topics: broker down
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: GroupCoordinator/5: Request metadata for 1 topic(s): broker down
%7|1692263746.489|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Broker changed state DOWN -> INIT
%7|1692263746.489|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263746.489|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Broker changed state INIT -> TRY_CONNECT
%7|1692263746.489|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263746.489|RECONNECT|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: Delaying next reconnect by 9999ms
%7|1692263746.489|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent MetadataRequest (v4, 34 bytes @ 0, CorrId 41822)
%7|1692263746.489|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received MetadataResponse (v4, 621 bytes, CorrId 41822, rtt 0.57ms)
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ===== Received metadata (for 1 requested topics): broker down =====
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ClusterId: lGqWfZ55TYy5KAJS2CHUbA, ControllerId: 2
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 6 brokers, 1 topics
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #0/6: b-5.test.kafka.us-east-1.amazonaws.com:9096 NodeId 5
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #1/6: b-1.test.kafka.us-east-1.amazonaws.com:9096 NodeId 1
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #2/6: b-4.test.kafka.us-east-1.amazonaws.com:9096 NodeId 4
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #3/6: b-6.test.kafka.us-east-1.amazonaws.com:9096 NodeId 6
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #4/6: b-2.test.kafka.us-east-1.amazonaws.com:9096 NodeId 2
%7|1692263746.489|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #5/6: b-3.test.kafka.us-east-1.amazonaws.com:9096 NodeId 3
%7|1692263746.490|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Topic #0/1: xtopic with 2 partitions
%7|1692263746.490|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 0 Leader 3
%7|1692263746.490|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 1 Leader 6
%7|1692263746.490|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 1/1 requested topic(s) seen in metadata
%7|1692263746.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [0]: broker is down: re-query
%7|1692263746.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [1]: broker is down: re-query
%7|1692263746.550|METADATA|rdkafka#consumer-1| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1692263746.550|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Request metadata for 1 topic(s): refresh unavailable topics
%7|1692263746.550|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent MetadataRequest (v4, 34 bytes @ 0, CorrId 41823)
%7|1692263746.551|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received MetadataResponse (v4, 621 bytes, CorrId 41823, rtt 1.27ms)
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ClusterId: lGqWfZ55TYy5KAJS2CHUbA, ControllerId: 2
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 6 brokers, 1 topics
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #0/6: b-5.test.kafka.us-east-1.amazonaws.com:9096 NodeId 5
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #1/6: b-1.test.kafka.us-east-1.amazonaws.com:9096 NodeId 1
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #2/6: b-4.test.kafka.us-east-1.amazonaws.com:9096 NodeId 4
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #3/6: b-6.test.kafka.us-east-1.amazonaws.com:9096 NodeId 6
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #4/6: b-2.test.kafka.us-east-1.amazonaws.com:9096 NodeId 2
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #5/6: b-3.test.kafka.us-east-1.amazonaws.com:9096 NodeId 3
%7|1692263746.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Topic #0/1: xtopic with 2 partitions
%7|1692263746.552|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 0 Leader 3
%7|1692263746.552|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 1 Leader 6
%7|1692263746.552|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 1/1 requested topic(s) seen in metadata
%7|1692263747.549|HEARTBEAT|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Heartbeat for group "ashu-consumer-test" generation id 11
%7|1692263747.549|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent HeartbeatRequest (v3, 93 bytes @ 0, CorrId 41824)
%7|1692263747.550|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received HeartbeatResponse (v3, 6 bytes, CorrId 41824, rtt 0.51ms)
%7|1692263747.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [0]: broker is down: re-query
%7|1692263747.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [1]: broker is down: re-query
%7|1692263747.550|METADATA|rdkafka#consumer-1| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1692263747.550|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Request metadata for 1 topic(s): refresh unavailable topics
%7|1692263747.550|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent MetadataRequest (v4, 34 bytes @ 0, CorrId 41825)
%7|1692263747.551|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received MetadataResponse (v4, 621 bytes, CorrId 41825, rtt 0.58ms)
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ClusterId: lGqWfZ55TYy5KAJS2CHUbA, ControllerId: 2
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 6 brokers, 1 topics
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #0/6: b-5.test.kafka.us-east-1.amazonaws.com:9096 NodeId 5
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #1/6: b-1.test.kafka.us-east-1.amazonaws.com:9096 NodeId 1
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #2/6: b-4.test.kafka.us-east-1.amazonaws.com:9096 NodeId 4
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #3/6: b-6.test.kafka.us-east-1.amazonaws.com:9096 NodeId 6
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #4/6: b-2.test.kafka.us-east-1.amazonaws.com:9096 NodeId 2
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #5/6: b-3.test.kafka.us-east-1.amazonaws.com:9096 NodeId 3
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Topic #0/1: xtopic with 2 partitions
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 0 Leader 3
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 1 Leader 6
%7|1692263747.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 1/1 requested topic(s) seen in metadata
%7|1692263748.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [0]: broker is down: re-query
%7|1692263748.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [1]: broker is down: re-query
%7|1692263748.550|METADATA|rdkafka#consumer-1| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1692263748.550|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Request metadata for 1 topic(s): refresh unavailable topics
%7|1692263748.550|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent MetadataRequest (v4, 34 bytes @ 0, CorrId 41826)
%7|1692263748.551|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received MetadataResponse (v4, 621 bytes, CorrId 41826, rtt 0.52ms)
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ClusterId: lGqWfZ55TYy5KAJS2CHUbA, ControllerId: 2
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 6 brokers, 1 topics
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #0/6: b-5.test.kafka.us-east-1.amazonaws.com:9096 NodeId 5
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #1/6: b-1.test.kafka.us-east-1.amazonaws.com:9096 NodeId 1
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #2/6: b-4.test.kafka.us-east-1.amazonaws.com:9096 NodeId 4
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #3/6: b-6.test.kafka.us-east-1.amazonaws.com:9096 NodeId 6
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #4/6: b-2.test.kafka.us-east-1.amazonaws.com:9096 NodeId 2
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #5/6: b-3.test.kafka.us-east-1.amazonaws.com:9096 NodeId 3
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Topic #0/1: xtopic with 2 partitions
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 0 Leader 3
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 1 Leader 6
%7|1692263748.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 1/1 requested topic(s) seen in metadata
%7|1692263749.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [0]: broker is down: re-query
%7|1692263749.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [1]: broker is down: re-query
%7|1692263749.550|METADATA|rdkafka#consumer-1| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1692263749.550|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Request metadata for 1 topic(s): refresh unavailable topics
%7|1692263749.550|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent MetadataRequest (v4, 34 bytes @ 0, CorrId 41827)
%7|1692263749.551|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received MetadataResponse (v4, 621 bytes, CorrId 41827, rtt 0.53ms)
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ClusterId: lGqWfZ55TYy5KAJS2CHUbA, ControllerId: 2
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 6 brokers, 1 topics
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #0/6: b-5.test.kafka.us-east-1.amazonaws.com:9096 NodeId 5
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #1/6: b-1.test.kafka.us-east-1.amazonaws.com:9096 NodeId 1
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #2/6: b-4.test.kafka.us-east-1.amazonaws.com:9096 NodeId 4
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #3/6: b-6.test.kafka.us-east-1.amazonaws.com:9096 NodeId 6
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #4/6: b-2.test.kafka.us-east-1.amazonaws.com:9096 NodeId 2
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #5/6: b-3.test.kafka.us-east-1.amazonaws.com:9096 NodeId 3
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Topic #0/1: xtopic with 2 partitions
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 0 Leader 3
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 1 Leader 6
%7|1692263749.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 1/1 requested topic(s) seen in metadata
%7|1692263750.549|HEARTBEAT|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Heartbeat for group "ashu-consumer-test" generation id 11
%7|1692263750.549|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent HeartbeatRequest (v3, 93 bytes @ 0, CorrId 41828)
%7|1692263750.550|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received HeartbeatResponse (v3, 6 bytes, CorrId 41828, rtt 0.48ms)
%7|1692263750.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [0]: broker is down: re-query
%7|1692263750.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [1]: broker is down: re-query
%7|1692263750.550|METADATA|rdkafka#consumer-1| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1692263750.550|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Request metadata for 1 topic(s): refresh unavailable topics
%7|1692263750.550|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent MetadataRequest (v4, 34 bytes @ 0, CorrId 41829)
%7|1692263750.551|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received MetadataResponse (v4, 621 bytes, CorrId 41829, rtt 0.49ms)
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ClusterId: lGqWfZ55TYy5KAJS2CHUbA, ControllerId: 2
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 6 brokers, 1 topics
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #0/6: b-5.test.kafka.us-east-1.amazonaws.com:9096 NodeId 5
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #1/6: b-1.test.kafka.us-east-1.amazonaws.com:9096 NodeId 1
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #2/6: b-4.test.kafka.us-east-1.amazonaws.com:9096 NodeId 4
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #3/6: b-6.test.kafka.us-east-1.amazonaws.com:9096 NodeId 6
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #4/6: b-2.test.kafka.us-east-1.amazonaws.com:9096 NodeId 2
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #5/6: b-3.test.kafka.us-east-1.amazonaws.com:9096 NodeId 3
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Topic #0/1: xtopic with 2 partitions
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 0 Leader 3
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 1 Leader 6
%7|1692263750.551|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 1/1 requested topic(s) seen in metadata
%7|1692263751.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [0]: broker is down: re-query
%7|1692263751.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [1]: broker is down: re-query
%7|1692263751.550|METADATA|rdkafka#consumer-1| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1692263751.550|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Request metadata for 1 topic(s): refresh unavailable topics
%7|1692263751.554|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent MetadataRequest (v4, 34 bytes @ 0, CorrId 41830)
%7|1692263751.555|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received MetadataResponse (v4, 621 bytes, CorrId 41830, rtt 0.70ms)
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ClusterId: lGqWfZ55TYy5KAJS2CHUbA, ControllerId: 2
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 6 brokers, 1 topics
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #0/6: b-5.test.kafka.us-east-1.amazonaws.com:9096 NodeId 5
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #1/6: b-1.test.kafka.us-east-1.amazonaws.com:9096 NodeId 1
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #2/6: b-4.test.kafka.us-east-1.amazonaws.com:9096 NodeId 4
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #3/6: b-6.test.kafka.us-east-1.amazonaws.com:9096 NodeId 6
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #4/6: b-2.test.kafka.us-east-1.amazonaws.com:9096 NodeId 2
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #5/6: b-3.test.kafka.us-east-1.amazonaws.com:9096 NodeId 3
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Topic #0/1: xtopic with 2 partitions
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 0 Leader 3
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 1 Leader 6
%7|1692263751.555|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 1/1 requested topic(s) seen in metadata
%7|1692263751.972|CONNECT|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: broker in state TRY_CONNECT connecting
%7|1692263751.972|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker changed state TRY_CONNECT -> CONNECT
%7|1692263751.972|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263751.973|CONNECT|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Connecting to ipv4#100.72.54.9:9096 (sasl_ssl) with socket 110
%7|1692263751.974|CONNECT|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Connected to ipv4#100.72.54.9:9096
%7|1692263751.974|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker changed state CONNECT -> SSL_HANDSHAKE
%7|1692263751.974|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263751.978|SSLVERIFY|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker SSL certificate verified
%7|1692263751.979|CONNECTED|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Connected (#1317)
%7|1692263751.979|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker changed state SSL_HANDSHAKE -> APIVERSION_QUERY
%7|1692263751.979|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263751.979|SEND|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Sent ApiVersionRequest (v3, 52 bytes @ 0, CorrId 116647)
%7|1692263751.979|RECV|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Received ApiVersionResponse (v3, 386 bytes, CorrId 116647, rtt 0.30ms)
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker API support:
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey Produce (0) Versions 0..8
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey Fetch (1) Versions 0..12
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey ListOffsets (2) Versions 0..5
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey Metadata (3) Versions 0..9
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey LeaderAndIsr (4) Versions 0..4
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey StopReplica (5) Versions 0..3
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey UpdateMetadata (6) Versions 0..6
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey ControlledShutdown (7) Versions 0..3
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey OffsetCommit (8) Versions 0..8
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey OffsetFetch (9) Versions 0..7
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey FindCoordinator (10) Versions 0..3
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey JoinGroup (11) Versions 0..7
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey Heartbeat (12) Versions 0..4
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey LeaveGroup (13) Versions 0..4
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey SyncGroup (14) Versions 0..5
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DescribeGroups (15) Versions 0..5
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey ListGroups (16) Versions 0..4
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey SaslHandshake (17) Versions 0..1
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey ApiVersion (18) Versions 0..3
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey CreateTopics (19) Versions 0..6
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DeleteTopics (20) Versions 0..5
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DeleteRecords (21) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey InitProducerId (22) Versions 0..4
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey OffsetForLeaderEpoch (23) Versions 0..3
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey AddPartitionsToTxn (24) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey AddOffsetsToTxn (25) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey EndTxn (26) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey WriteTxnMarkers (27) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey TxnOffsetCommit (28) Versions 0..3
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DescribeAcls (29) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey CreateAcls (30) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DeleteAcls (31) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DescribeConfigs (32) Versions 0..3
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey AlterConfigs (33) Versions 0..1
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey AlterReplicaLogDirs (34) Versions 0..1
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DescribeLogDirs (35) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey SaslAuthenticate (36) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey CreatePartitions (37) Versions 0..3
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey CreateDelegationToken (38) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey RenewDelegationToken (39) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey ExpireDelegationToken (40) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DescribeDelegationToken (41) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DeleteGroups (42) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey ElectLeadersRequest (43) Versions 0..2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey IncrementalAlterConfigsRequest (44) Versions 0..1
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey AlterPartitionReassignmentsRequest (45) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey ListPartitionReassignmentsRequest (46) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey OffsetDeleteRequest (47) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DescribeClientQuotasRequest (48) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey AlterClientQuotasRequest (49) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey DescribeUserScramCredentialsRequest (50) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey AlterUserScramCredentialsRequest (51) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey AlterIsrRequest (56) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: ApiKey UpdateFeaturesRequest (57) Versions 0..0
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature MsgVer1: Produce (2..2) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature MsgVer1: Fetch (2..2) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature MsgVer1
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature MsgVer2: Produce (3..3) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature MsgVer2: Fetch (4..4) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature MsgVer2
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature ApiVersion: ApiVersion (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature ApiVersion
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature BrokerGroupCoordinator: FindCoordinator (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature BrokerGroupCoordinator
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature BrokerBalancedConsumer: FindCoordinator (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature BrokerBalancedConsumer
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature ThrottleTime: Produce (1..2) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature ThrottleTime: Fetch (1..2) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature ThrottleTime
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature Sasl: JoinGroup (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature Sasl
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature SaslHandshake: SaslHandshake (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature SaslHandshake
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature LZ4: FindCoordinator (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature LZ4
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature OffsetTime: ListOffsets (1..1) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature OffsetTime
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature IdempotentProducer: InitProducerId (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature IdempotentProducer
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature ZSTD: Produce (7..7) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature ZSTD: Fetch (10..10) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature ZSTD
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature SaslAuthReq: SaslHandshake (1..1) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Feature SaslAuthReq: SaslAuthenticate (0..0) supported by broker
%7|1692263751.979|APIVERSION|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Enabling feature SaslAuthReq
%7|1692263751.979|AUTH|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Auth in state APIVERSION_QUERY (handshake supported)
%7|1692263751.979|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker changed state APIVERSION_QUERY -> AUTH_HANDSHAKE
%7|1692263751.979|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263751.979|SEND|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Sent SaslHandshakeRequest (v1, 36 bytes @ 0, CorrId 116648)
%7|1692263751.980|RECV|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Received SaslHandshakeResponse (v1, 21 bytes, CorrId 116648, rtt 0.24ms)
%7|1692263751.980|SASLMECHS|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker supported SASL mechanisms: SCRAM-SHA-512
%7|1692263751.980|AUTH|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Auth in state AUTH_HANDSHAKE (handshake supported)
%7|1692263751.980|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker changed state AUTH_HANDSHAKE -> AUTH_REQ
%7|1692263751.980|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263751.980|SASL|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Initializing SASL client: service name kafka, hostname b-6.test.kafka.us-east-1.amazonaws.com, m
echanisms SCRAM-SHA-512, provider SCRAM (builtin)
%7|1692263751.980|SASLSCRAM|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: SASL SCRAM client in state client-first-message
%7|1692263751.980|SASL|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Send SASL Kafka frame to broker (60 bytes)
%7|1692263751.980|SEND|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Sent SaslAuthenticateRequest (v0, 85 bytes @ 0, CorrId 116649)
%7|1692263751.980|RECV|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Received SaslAuthenticateResponse (v0, 113 bytes, CorrId 116649, rtt 0.58ms)
%7|1692263751.980|SASL|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Received SASL frame from broker (105 bytes)
%7|1692263751.980|SASLSCRAM|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: SASL SCRAM client in state server-first-message
%7|1692263751.989|SASL|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Send SASL Kafka frame to broker (189 bytes)
%7|1692263751.990|SEND|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Sent SaslAuthenticateRequest (v0, 214 bytes @ 0, CorrId 116650)
%7|1692263751.990|RECV|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Received SaslAuthenticateResponse (v0, 112 bytes, CorrId 116650, rtt 0.38ms)
%7|1692263751.990|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCR
AM-SHA-512 (after 10ms in state AUTH_REQ) (_AUTHENTICATION): identical to last error: error log suppressed
%7|1692263751.990|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker changed state AUTH_REQ -> DOWN
%7|1692263751.990|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263751.990|BUFQ|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Purging bufq with 0 buffers
%7|1692263751.990|BUFQ|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Purging bufq with 0 buffers
%7|1692263751.990|BUFQ|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Updating 0 buffers on connection reset
%7|1692263751.990|METADATA|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: Requesting metadata for 1/1 topics: broker down
%7|1692263751.990|METADATA|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: GroupCoordinator/5: Request metadata for 1 topic(s): broker down
%7|1692263751.990|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker changed state DOWN -> INIT
%7|1692263751.990|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263751.990|STATE|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Broker changed state INIT -> TRY_CONNECT
%7|1692263751.990|BROADCAST|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: Broadcasting state change
%7|1692263751.990|RECONNECT|rdkafka#consumer-1| [thrd:sasl_ssl://b-6.test.kafka.us-east-1.am]: sasl_ssl://b-6.test.kafka.us-east-1.amazonaws.com:9096/6: Delaying next reconnect by 9999ms
%7|1692263751.990|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent MetadataRequest (v4, 34 bytes @ 0, CorrId 41831)
%7|1692263751.991|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received MetadataResponse (v4, 621 bytes, CorrId 41831, rtt 0.68ms)
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ===== Received metadata (for 1 requested topics): broker down =====
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ClusterId: lGqWfZ55TYy5KAJS2CHUbA, ControllerId: 2
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 6 brokers, 1 topics
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #0/6: b-5.test.kafka.us-east-1.amazonaws.com:9096 NodeId 5
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #1/6: b-1.test.kafka.us-east-1.amazonaws.com:9096 NodeId 1
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #2/6: b-4.test.kafka.us-east-1.amazonaws.com:9096 NodeId 4
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #3/6: b-6.test.kafka.us-east-1.amazonaws.com:9096 NodeId 6
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #4/6: b-2.test.kafka.us-east-1.amazonaws.com:9096 NodeId 2
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #5/6: b-3.test.kafka.us-east-1.amazonaws.com:9096 NodeId 3
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Topic #0/1: xtopic with 2 partitions
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 0 Leader 3
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 1 Leader 6
%7|1692263751.991|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 1/1 requested topic(s) seen in metadata
%7|1692263752.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [0]: broker is down: re-query
%7|1692263752.550|QRYLEADER|rdkafka#consumer-1| [thrd:main]: Topic xtopic [1]: broker is down: re-query
%7|1692263752.550|METADATA|rdkafka#consumer-1| [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
%7|1692263752.550|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Request metadata for 1 topic(s): refresh unavailable topics
%7|1692263752.551|SEND|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Sent MetadataRequest (v4, 34 bytes @ 0, CorrId 41832)
%7|1692263752.553|RECV|rdkafka#consumer-1| [thrd:GroupCoordinator]: GroupCoordinator/5: Received MetadataResponse (v4, 621 bytes, CorrId 41832, rtt 2.56ms)
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: ClusterId: lGqWfZ55TYy5KAJS2CHUbA, ControllerId: 2
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 6 brokers, 1 topics
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #0/6: b-5.test.kafka.us-east-1.amazonaws.com:9096 NodeId 5
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #1/6: b-1.test.kafka.us-east-1.amazonaws.com:9096 NodeId 1
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #2/6: b-4.test.kafka.us-east-1.amazonaws.com:9096 NodeId 4
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #3/6: b-6.test.kafka.us-east-1.amazonaws.com:9096 NodeId 6
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #4/6: b-2.test.kafka.us-east-1.amazonaws.com:9096 NodeId 2
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Broker #5/6: b-3.test.kafka.us-east-1.amazonaws.com:9096 NodeId 3
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: Topic #0/1: xtopic with 2 partitions
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 0 Leader 3
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: Topic xtopic partition 1 Leader 6
%7|1692263752.553|METADATA|rdkafka#consumer-1| [thrd:main]: GroupCoordinator/5: 1/1 requested topic(s) seen in metadata 
emasab
haoboliu66
vishwa-trulioo
Description
We are using MSK(AWS Apache kafka) cluster with SASL-SCRAM authentication, here credentials get auto rotated after 90 days when our application is running. There is no impact of password change until unless by any/various reason client try to connect to new broker(essentially reestablishing the TCP connection). At that time authentication happens and gives "SASL authentication error".
%7|1692263746.489|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (after 11ms in state AUTH_REQ) (_AUTHENTICATION): identical to last error: error log suppressedOur issue here is that this error doesn't invoke the error handler which we have assigned to Consumer so we can rebuild the consumer with latest password in case of authentication failure. It just runs in loop and call never come back to client and remain in underlying kafka library.
Just for explaining how we read password: If we or Confluent-kafka-dotnet lib will try to read password from
sasl.passwordkey in config dictionary - we will always go to secrets manager and fetch the latest password. We have overridden IDictionary so any call for Get with keysasl.passwordwill go to secrets managerHow to reproduce
%7|1692263746.489|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://b-3.test.kafka.us-east-1.am]: sasl_ssl://b-3.test.kafka.us-east-1.amazonaws.com:9096/3: SASL authentication error: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512 (after 11ms in state AUTH_REQ) (_AUTHENTICATION): identical to last error: error log suppressedHere if we get this error some how in consumer handler we can rebuild our consumer. MSK doesn't allow us to set connections.max.reauth.ms and gives "Key 'connections.max.reauth.ms' is not supported by at least one Apache Kafka version. Key checked against versions: [2.8.1.2, 2.6.2, 2.7.1, 2.8.0, 2.6.3, 2.7.2, 2.8.1, 2.5.1, 2.6.0, 2.6.1, 2.7.0, 2.3.1 , 2.2.1, 2.4.1.1, 3.3.2, 3.1.1, 3.2.0, 3.3.1, 3.4.0] error.
We are looking to find someway by which Auth error comes back to client side so we can rebuild the consumer.