search

bsm / sarama-cluster

Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9 [DEPRECATED]
MIT License
1.01k stars 222 forks source link

how to limit number of partitions assigned to consumer #283

Closed nitin302 closed 5 years ago

nitin302 commented 5 years ago

For Kafka, partitions provide a way for consumption to scale. Hence, it is necessary that consumers needs to scale horizontally as well.

1) Is there any way to limit number of partitions per consumer, so that we can add more consumers so that all partitions are assigned to one of the consumers (total consumers * partitions per consumer = total partitions of the topic)? How can I do this?

2) What should the configuration of the node in this case? What logic can be used for size a consumer for a given number of partitions?

dim commented 5 years ago

@nitin302 I suggest that you look into https://github.com/Shopify/sarama/pull/1099 which has been merged and released, as sarama-cluster is now deprecated. You can configure a custom https://godoc.org/github.com/Shopify/sarama#BalanceStrategy which can plan the partition allocation to consumers based on a consumer's UserData. Hope that helps!

Shan4Kites commented 5 years ago

@dim can you please add "deprecated" in the github description for this repo? it will help the new users.