Multiple consumers in consumer group breaks all consumers in group?

[ngalchemist]

Describe the bug I have a Nodejs app running in a Docker container that creates multiple consumers on startup. Each one of these consumers is for a a different topic with a single partition. All of these topics are added to the same group. When the second consumer is added to the group, I receive the logs below and neither consumer starts consuming messages. The log group in the example is 'log-sub-group' and the two topics are 'tracing.enterprise.connectionbus-update.v1' and 'tracing.enterprise.microflowengine.v1'.

{"level":"INFO","timestamp":"2023-08-04T21:58:33.820Z","logger":"kafkajs","message":"[Consumer] Starting","groupId":"log-sub-group"}
{"level":"ERROR","timestamp":"2023-08-04T21:58:38.836Z","logger":"kafkajs","message":"[Connection] Response Heartbeat(key: 12, version: 3)","broker":"kafka:9093","clientId":"log-subscriber-consumer","error":"The group is rebalancing, so a rejoin is needed","correlationId":9,"size":10}
{"level":"WARN","timestamp":"2023-08-04T21:58:38.837Z","logger":"kafkajs","message":"[Runner] The group is rebalancing, re-joining","groupId":"log-sub-group","memberId":"log-subscriber-consumer-f40e98b5-4519-4755-b89e-8b0d2976d9eb","error":"The group is rebalancing, so a rejoin is needed"}
{"level":"WARN","timestamp":"2023-08-04T21:58:38.841Z","logger":"kafkajs","message":"[ConsumerGroup] Consumer group received unsubscribed topics","groupId":"log-sub-group","generationId":6,"memberId":"log-subscriber-consumer-022c92b6-65dd-4b29-8ac5-22c10e71404a","assignedTopics":["tracing.enterprise.connectionbus-update.v1"],"topicsSubscribed":["tracing.enterprise.microflowengine.v1"],"topicsNotSubscribed":["tracing.enterprise.connectionbus-update.v1"],"helpUrl":"https://kafka.js.org/docs/faq#why-am-i-receiving-messages-for-topics-i-m-not-subscribed-to"}
{"level":"INFO","timestamp":"2023-08-04T21:58:38.842Z","logger":"kafkajs","message":"[ConsumerGroup] Consumer has joined the group","groupId":"log-sub-group","memberId":"log-subscriber-consumer-022c92b6-65dd-4b29-8ac5-22c10e71404a","leaderId":"log-subscriber-consumer-f40e98b5-4519-4755-b89e-8b0d2976d9eb","isLeader":false,"memberAssignment":{},"groupProtocol":"RoundRobinAssigner","duration":5022}
{"level":"INFO","timestamp":"2023-08-04T21:58:38.842Z","logger":"kafkajs","message":"[ConsumerGroup] Consumer has joined the group","groupId":"log-sub-group","memberId":"log-subscriber-consumer-f40e98b5-4519-4755-b89e-8b0d2976d9eb","leaderId":"log-subscriber-consumer-f40e98b5-4519-4755-b89e-8b0d2976d9eb","isLeader":true,"memberAssignment":{},"groupProtocol":"RoundRobinAssigner","duration":5}

To Reproduce

import { Kafka } from 'kafkajs';

const kafka = new Kafka(); // Real config here
let consumers = [{Topic: 'A', Group: 'my-group'}, {Topic: 'B', Group: 'my-group'}];
for (let consumer of consumers) {
   const consumer = kafka.consumer({ groupId: consumer.Group });
   await consumer.connect();
   await consumer.subscribe({ topics: [consumer.Topic] });
   await consumer.run({ eachMessage: () => console.log('Message received!') });
}

Expected behavior The consumer should be able to run two consumers for two different topics in the same group.

Observed behavior I receive the error in the description above when running the code.

Environment:

• OS: [linux/amd64]
• KafkaJS version [2.2.4]
• Kafka version [e.g. 3.4.0]
• NodeJS version [e.g. 18.16.0]

Additional context I'm not sure if KafkaJs doesn't support this or if my implementation is incorrect. The docs here say that the warning "likely means that some members of your consumer group are subscribed to some topics, and some other members of the group are subscribed to a different set of topics" which makes me think I'm only supposed to be consuming for one topic on all consumers in a consumer group. But I haven't had to do this with the Java consumer implementations I've built.

[BonesyWonesy]

Also running in to this.

About to rewrite all my stuff in Java and ditch this library.

[leppaott]

This is normal while deploying a new version of an application where the new version subscribes to a new topic, and the warning will go away once the group stabilizes on a single version.

Yeah seems only the last subscribed topics remain i.e. makes sense when rolling new versions in.

Fixed by appending topic names into groupId. Don't think that's a huge workaround even if Kafka could work with having different topic subscribers on the same group.