ConsumerGroup does not comsume data from all partitions

[emmaccacca]

Questions?

I use HighLevelProducer to produce keyed message and then I use ConsumerGroup to consume messages. But I only get message of one or two partitions every time I restart my service.

I have a question. How can I ensure the order of messages ussing HighLevelProducer and ConsumerGroup?

Bug Report

Environment

• Node version: 8.9.4
• Kafka-node version: 2.11
• Kafka version: 1.0

For specific cases also provide

• Number of Brokers: 3
• Number partitions for topic: 4

Include Sample Code to reproduce behavior

const client = new Client({
    kafkaHost
});
const producer = new Producer(client);

producer.send([{
    topic: 'topic',
    messages: new KeyedMessage('key', 'message')
}], (err, data) => {
    if (err) {
        console.error(err);
    }
});

const consumerOptions = {
    kafkaHost,
    groupId: 'log',
    autoCommit: false,
    fromOffset: 'earliest'
};

const consumer = new Consumer(consumerOptions, ['topic']);

Include output with Debug turned on

Thanks for your contribution!

[aikar]

As I responded on https://github.com/SOHU-Co/kafka-node/issues/904 - Considering your import is named "Consumer", and not "ConsumerGroup", I assume you imported from lib/consumer.js

If this is the case, you need to use ConsumerGroup (lib/consumerGroup.js) instead, if you want all partitions to be automatically assigned. Consumer requires manually specifying partitions.

@hyperlink I can see why this is a source of confusion considering the docs list Consumer first, but still uses a group ID in config.

Maybe need to update docs to say 'You probably want '

[hyperlink]

You're right @aikar there's certainly improvement opportunities in the docs.