SOHU-Co / kafka-node

Node.js client for Apache Kafka 0.8 and later.
MIT License
2.67k stars 628 forks source link

consumers in consumer group gets no partitions assigned to them. #1363

Open dev-websplanet opened 4 years ago

dev-websplanet commented 4 years ago

Environment Node version: 10.15.3 Kafka-node version: 5.0.0 Kafka version (single node): 2.3.0 Number of Brokers: 1 Number of partitions for topic: 1

my app consists on micro services, and each micro service can have 1 - 3 instances running (only one should receive each message). so if i understood the correct usage of kafka consumer groups, when each ms is started it creates a consumer group with the groupId of the ms name (unique per ms), each topic is added with 1 partition, and i figured only one consumer will consume the messages from the only partition, and if the consumer leaves the group some other consumer will start consuming from that partition.

sometimes the consumer group is created and the topics are added but i can see when i run commands from the kafka docker that no consumer is assigned to the partition, and i cant find the reason.

command: kafka-consumer-groups.sh --describe --group ums --bootstrap-server localhost:9092

valid response:

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID

resConfig_update 0 0 0 0 ums-consumer-1575794029875-4f78bb47-84f9-4d75-bede-3b838802723f /172.16.103.20 ums-consumer-1575794029875

reseller_delete 0 0 0 0 ums-consumer-1575794029875-4f78bb47-84f9-4d75-bede-3b838802723f /172.16.103.20 ums-consumer-1575794029875

reseller_create 0 0 0 0 ums-consumer-1575794029875-4f78bb47-84f9-4d75-bede-3b838802723f /172.16.103.20 ums-consumer-1575794029875

sometimes i get this:

TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID -

resConfig_update 0 0 0 0 - - -

reseller_delete 0 0 0 0 - - -

reseller_create 0 0 0 0 - - -

im seding a SyncGroupRequest from the consumer to debug the the problem and i can see the consumer has no partitions assigned to him. this is the consumerGroup creation:

` let options = { id: "ums-consumer-${Date.now()}", clientId: "ums-consumer-${Date.now()}", kafkaHost: host, groupId: 'ums', protocol: ['roundrobin'], idleConnection: 60000, fromOffset: 'earliest'};

let ConsumerGroup = kafka.ConsumerGroup; let consumer = new ConsumerGroup(options, ['reseller_create', 'reseller_delete', 'resConfig_update']); `

i saw a post about a similar problem but the solutions did not work, and the one who post it just changed clients eventually. post: https://github.com/SOHU-Co/kafka-node/issues/339

0xslipk commented 4 years ago

@dev-websplanet maybe is related to this issue #1355