search

Blizzard / node-rdkafka

Node.js bindings for librdkafka
MIT License
2.08k stars 391 forks source link

Consuming messages from each partition separately #1063

Open everhardt opened 7 months ago

everhardt commented 7 months ago

A Consumer that is subscribed to multiple partitions cannot control the mix of messages it consumes from each partition. There are various reasons why controlling that is relevant, see eg. #719 and https://github.com/confluentinc/librdkafka/wiki/FAQ#what-are-partition-queues-and-why-are-some-partitions-slower-than-others.

That last link shows that librdkafka supports consuming from each partition separately. It would be great if that implementation is available in node-rdkafka as well. An example could be a PartitionConsumer class, that does not have the normal .consume methods, but .consume(topic, partition, number, callback).

martijnimhoff commented 6 months ago

We're working on a PR: https://github.com/withthegrid/node-rdkafka/pull/2

pveller commented 2 months ago

re: #719 . node-rdkafka would probably need to migrate to using rd_kafka_queue_new / rd_kafka_consume_queue. You can't do better if you are on rd_kafka_consume

https://github.com/confluentinc/librdkafka/wiki/Consuming-from-multiple-topics-partitions-from-a-single-thread