Environment
Node version: 10.15.3
Kafka-node version: 5.0.0
Kafka version (single node): 2.3.0
Number of Brokers: 1
Number of partitions for topic: 1
my app consists on micro services, and each micro service can have 1 - 3 instances running (only one should receive each message).
so if i understood the correct usage of kafka consumer groups, when each ms is started it creates a consumer group with the groupId of the ms name (unique per ms), each topic is added with 1 partition, and i figured only one consumer will consume the messages from the only partition, and if the consumer leaves the group some other consumer will start consuming from that partition.
sometimes the consumer group is created and the topics are added but i can see when i run commands from the kafka docker that no consumer is assigned to the partition, and i cant find the reason.
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID -
resConfig_update 0 0 0 0 - - -
reseller_delete 0 0 0 0 - - -
reseller_create 0 0 0 0 - - -
im seding a SyncGroupRequest from the consumer to debug the the problem and i can see the consumer has no partitions assigned to him.
this is the consumerGroup creation:
Environment Node version: 10.15.3 Kafka-node version: 5.0.0 Kafka version (single node): 2.3.0 Number of Brokers: 1 Number of partitions for topic: 1
my app consists on micro services, and each micro service can have 1 - 3 instances running (only one should receive each message). so if i understood the correct usage of kafka consumer groups, when each ms is started it creates a consumer group with the groupId of the ms name (unique per ms), each topic is added with 1 partition, and i figured only one consumer will consume the messages from the only partition, and if the consumer leaves the group some other consumer will start consuming from that partition.
sometimes the consumer group is created and the topics are added but i can see when i run commands from the kafka docker that no consumer is assigned to the partition, and i cant find the reason.
command: kafka-consumer-groups.sh --describe --group ums --bootstrap-server localhost:9092
valid response:
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
resConfig_update 0 0 0 0 ums-consumer-1575794029875-4f78bb47-84f9-4d75-bede-3b838802723f /172.16.103.20 ums-consumer-1575794029875
reseller_delete 0 0 0 0 ums-consumer-1575794029875-4f78bb47-84f9-4d75-bede-3b838802723f /172.16.103.20 ums-consumer-1575794029875
reseller_create 0 0 0 0 ums-consumer-1575794029875-4f78bb47-84f9-4d75-bede-3b838802723f /172.16.103.20 ums-consumer-1575794029875
sometimes i get this:
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID -
resConfig_update 0 0 0 0 - - -
reseller_delete 0 0 0 0 - - -
reseller_create 0 0 0 0 - - -
im seding a SyncGroupRequest from the consumer to debug the the problem and i can see the consumer has no partitions assigned to him. this is the consumerGroup creation:
` let options = { id: "ums-consumer-${Date.now()}", clientId: "ums-consumer-${Date.now()}", kafkaHost: host, groupId: 'ums', protocol: ['roundrobin'], idleConnection: 60000, fromOffset: 'earliest'};
let ConsumerGroup = kafka.ConsumerGroup; let consumer = new ConsumerGroup(options, ['reseller_create', 'reseller_delete', 'resConfig_update']); `
i saw a post about a similar problem but the solutions did not work, and the one who post it just changed clients eventually. post: https://github.com/SOHU-Co/kafka-node/issues/339