search

tulios / kafkajs

A modern Apache Kafka client for node.js
https://kafka.js.org
MIT License
3.71k stars 525 forks source link

While using eachBatch(), the consumer isn't processing messages from each batch and just calling multiple batches #1700

Open anupamd2-zluri opened 3 months ago

anupamd2-zluri commented 3 months ago

Kafka Setup

2 Broker URLs 1 Topic with 4 Partitions - workflow-executions 1 Consumer Group (raw-data-consumer) with 4 Consumers (Kubernetes PODs)

Currently we are using eachMessage() to process the messages from the Brokers but we are not able to achieve the required throughput.

We are doing a POC of using eachBatch() to process the messages in batches.

While using eachBatch(), we found out that the messages from each batch isn't being processed but the handler function keeps on calling the eachBatch(). Is this the expected behaviour?

image image image
shardul-cometchat commented 2 weeks ago

I am facing a similar issue for eachMessage and the only solution I could find till now is setting partitionsConsumedConcurrently to 1 based on this: https://github.com/tulios/kafkajs/issues/1492