Consumer consumes a message twice!

vking34 commented 4 years ago

Describe the bug

I just follow the document to set up a consumer. But the consumer consume a message twice.

How to Reproduce

import KafkaAvro from 'kafka-node-avro';
import sendAuctionResult from '../sockets/newAuctionResultEvent';

const settings = {
    kafka: {
        kafkaHost: process.env.KAFKA_BOOTSTRAP_SERVERS
    },
    schema: {
        registry: process.env.SCHEMA_REGISTRY_URL,
        topics: [{ name: 'chozoi.socket.auction.result' }]
    }
}

KafkaAvro.init(settings)
    .then((kafka) => {
        let biddingConsumer = kafka.addConsumer(
            'chozoi.socket.auction.result',
            {
                groupId: 'auction-event-consumer-node',
                fromOffset: 'latest',
                commitOffsetsOnFirstJoin: true
            }
        );

        biddingConsumer.on('message', auctionResultMessage => {
            console.log("auction result ", auctionResultMessage.value.id);
            sendAuctionResult(auctionResultMessage.value);
        });
    }).catch((_e) => {
        console.log(_e);
    });

export default KafkaAvro;

And then i got the following result:

auction result  3828
auction result  3828

Library Dependencies

 "dependencies": {
        "axios": "^0.20.0",
        "body-parser": "^1.19.0",
        "cuid": "^2.1.8",
        "express": "^4.17.1",
        "http": "0.0.1-security",
        "kafka-node": "^5.0.0",
        "kafka-node-avro": "^4.3.0",
        "mongoose": "^5.10.7",
        "path": "^0.12.7",
        "redis": "^3.0.2",
        "socket.io": "^2.3.0",
        "socket.io-redis": "^5.4.0"
    },
    "devDependencies": {
        "@types/axios": "^0.14.0",
        "@types/express": "^4.17.8",
        "@types/mongoose": "^5.7.36",
        "@types/node": "^14.11.1",
        "@types/socket.io": "^2.1.11",
        "cors": "^2.8.5",
        "morgan": "^1.10.0",
        "nodemon": "^2.0.4",
        "ts-node": "^9.0.0",
        "typescript": "^4.0.3",
        "util": "^0.12.3"
    }

narcisoguillen commented 3 years ago

Hi @vking34 thanks for using kafka-node-avro, I tried to replicate the same issue with no luck, nevertheless I made some research and it might be unfortunately an active/open issue on kafka-node library.

vking34 commented 3 years ago

Hi @vking34 thanks for using kafka-node-avro, I tried to replicate the same issue with no luck, nevertheless I made some research and it might be unfortunately an active/open issue on kafka-node library.

Related issues. SOHU-Co/kafka-node#1042 SOHU-Co/kafka-node#1318 SOHU-Co/kafka-node#1434

Thank for your reply.

vking34 commented 3 years ago

@narcisoguillen I used conduktor tool to consume and check the topic. The topic works normally. But with the above settings, my consumer still consumes a message twice. Could you give me a sample code for a consumer?

narcisoguillen / kafka-node-avro

Consumer consumes a message twice! #25