search

spring-cloud / spring-cloud-stream-binder-kafka

Spring Cloud Stream binders for Apache Kafka and Kafka Streams
Apache License 2.0
331 stars 302 forks source link

A Topic lost a subscription node during run time #1183

Open hero-zhanghao opened 2 years ago

hero-zhanghao commented 2 years ago

Spring boot Version : 2.5.0 Spring Cloud Version ： 2020.0.3

I used Spring-Cloud-stream-binder - Kafka and Spring-cloud-stream-binder - Kafka-Streams for kafka production and consumption in the project.

In one project, I subscribed to N topics.

Two nodes were started for service using load balancing.

During run time, it was suddenly discovered that one of the topics had no subscription nodes. This results in messages being backlogged and lost.

I have to restart these service nodes before I can subscribe to this Topic again.

What is the cause of this, or is there any way to help find some clues.

And is there a way to check at run time so that topics that have lost subscriptions can be re-subscribed?

hero-zhanghao commented 2 years ago

I found such a log, whether related to the problem?

Member consumer-8-a776d75e-cdab-47b6-bd2a-b5f68117a7b5 sending LeaveGroup request to coordinator 127.0.0.1:9092 (id: 2147483646 rack: null) due to consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.

NajeebArif commented 2 years ago

Can you please share the code which you are using to consume the records? Most probably issue seems with the max poll records. Also keep an eye for the heartbeat which consumer sends to the coordinator because that could also cause cosumers being treated as dead and they will be removed.