tulios / kafkajs

A modern Apache Kafka client for node.js
https://kafka.js.org
MIT License
3.76k stars 529 forks source link

[Question] KafkaJS handling of failed heartbeat calls #1719

Open krumft opened 3 weeks ago

krumft commented 3 weeks ago

Hi Team,

I am opening this issue just for the sake of asking a question. When the KafkaJS library performs a heartbeat call to the Kafka cluster, and receives an error that the consumer group is being rebalanced:

  1. will KafkaJS pause the consuming for that consumer? (and presumably resume it when this consumer successfully rejoins the group) - this would make sense as it is no longer clear for which topic partitions the consumer is responsible
  2. will KafkaJS start preparing the consumer to issue a ReJoin request to Kafka?

Looking at https://github.com/tulios/kafkajs/blob/master/src/consumer/runner.js#L207-L216 it seems that neither of these two happen.

The reasons behind these questions of mine are the following two:

  1. We are seeing evidence of both 1. and 2. happening in the case of the Java consumer;
  2. If we manually implement 1. and 2. to happen during our application-level handling of heartbeat it seems we can solve some nasty issues of constantly seeing rebalancing.

Thanks a lot for your time, Krum.

krumft commented 3 weeks ago

Related question: https://github.com/tulios/kafkajs/issues/1645