search

SOHU-Co / kafka-node

Node.js client for Apache Kafka 0.8 and later.
MIT License
2.66k stars 628 forks source link

Duplicate messages received #1434

Open jransome opened 4 years ago

jransome commented 4 years ago

We are using a v5.0.0 ConsumerGroup (node 12) to consume messages and manually commit them once we have finished processing them. On each message, we call pause() on the consumer, process the message (which can take on average 700ms +/-200ms), call commit() and then resume() the consumer. However this results in the occasional receipt of duplicate messages under certain conditions where there is a backlog (at least >100 messages) where the consumer will emit x number of messages between commits followed by a repetition of the last n number of messages in that batch of x. We are using the following configuration that may be relevant:

fetchMaxBytes: 524288 fetchMinBytes: 10 * 1024 maxTickMessages: 50 (also up to 100) autoCommitIntervalMs: 1000 autoCommit: false

Is there something that might explain this repetition of the last n number of messages between commits?

vking34 commented 4 years ago

@jransome I have the issue. Did you solve it?