When the broker stops, the subscriber consumer lag

[yp19910928]

Description

I have a Kafka cluster with 5 brokers and 5 controllers. The 5 brokers are grouped as H: (H1/H2), N: (N1/N2), and D: (D1). I have subscriber programs running on the VMs of H1/H2/N1/N2 brokers, which subscribe to the same topic, and send 1 message per second to the topic. When I stop the N1/N2/D1 broker services, the publisher program can publish messages normally, and the subscriber programs on the H1/H2 VMs are also normal. My expectation is that the subscriber programs on the N1/N2 VMs can also subscribe to messages normally, but in reality, the subscriber programs on the N1/N2 VMs can only consume partial Partition messages, and some Partition messages have consumer lag.

H1:5245 H2:5246 N1:5247 N2:5248 D1:52104

How to reproduce

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [x] Confluent.Kafka nuget version.
• [x] Apache Kafka version.
• [ ] Client configuration.
• [x] Operating system.
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [ ] Critical issue.

[yp19910928]

Kafka Broker Config: num.partitions=4 offsets.topic.replication.factor=4 transaction.state.log.replication.factor=4 transaction.state.log.min.isr=2 default.replication.factor=4 min.insync.replicas=2

Apache Kafka version: 3.3.1

Operating system. Linux photon 4.19.132-6.ph3 #1-photon SMP Sun Aug 9 02:22:28 UTC 2020 x86_64 GNU/Linux

Confluent.Kafka nuget version 2.3.0