I have a use case where I would like to parallelize consuming from a single rdkafka consumer. I have a static set of partitions that I would like to subdivide across a smaller set of threads (think 12 partitions, 3 or 4 threads). librdkafka supports this ( see: https://github.com/edenhill/librdkafka/wiki/Consuming-from-multiple-topics-partitions-from-a-single-thread ), but rdkafka doesn't expose it. The split_partition_queue method only supports splitting off a single partition at a time. I would instead like to see a method that I can pass a TopicPartitionList to, and get a message queue for all of the partitions.
Is this something that would be of interest to the project? If so, I can work on implementing it.
benesch
commented
3 years ago
I have a use case where I would like to parallelize consuming from a single rdkafka consumer. I have a static set of partitions that I would like to subdivide across a smaller set of threads (think 12 partitions, 3 or 4 threads). librdkafka supports this ( see: https://github.com/edenhill/librdkafka/wiki/Consuming-from-multiple-topics-partitions-from-a-single-thread ), but
rdkafkadoesn't expose it. The split_partition_queue method only supports splitting off a single partition at a time. I would instead like to see a method that I can pass aTopicPartitionListto, and get a message queue for all of the partitions.Is this something that would be of interest to the project? If so, I can work on implementing it.