Description
The number of consumers created for a single topic is calculated using:
int numOfThreads = Math.min(numOfConsumers, listPartitionInfo.size());
But because the kafka consumer groups are designed in a way that there is actual use for having more consumers than available topic partitions, it is wiser to not limit the number of consumers based on available topic partitions.
The additional in-active consumers can replace active consumers when they fail and/or a consumer group rebalance happens.
Tiihott
commented
2 months ago
Description The number of consumers created for a single topic is calculated using:
int numOfThreads = Math.min(numOfConsumers, listPartitionInfo.size());But because the kafka consumer groups are designed in a way that there is actual use for having more consumers than available topic partitions, it is wiser to not limit the number of consumers based on available topic partitions. The additional in-active consumers can replace active consumers when they fail and/or a consumer group rebalance happens.