SOHU-Co / kafka-node

Node.js client for Apache Kafka 0.8 and later.
MIT License
2.66k stars 627 forks source link

Kafka Consumer not consuming from random topics abruptly #831

Open LeeFung opened 6 years ago

LeeFung commented 6 years ago

Question / Bug?

I am currently running 2 set ups:

A) 9 servers sharing the 9 partitions B) 2 servers consuming from 9 partitions each I am running one consumer to consumer from multiple topics on each server.

My observation is that setup A is very stable and it doesn't have any missing messages nor does it have any incident where it stops consuming messages.

While for setup B, there is a possibility that messages from a few topics are not being consumed anymore, causing a break in the data flow. However, messages from other topics are being consumed normally.

There were no errors reported (thus no logs) and the consumer just silently stop consuming from a few random topics.

Would appreciate a response if anyone have any idea or require more information.

Thank you!

Environment

LeeFung commented 6 years ago

Bumping this. Anyone have similar experiences? Consumers just suddenly stop consuming from the assigned group.

hyperlink commented 6 years ago

There has been some fixes related to recovery. Could you test again using the latest version?