prevent n+1 consumer from joining group with n partitions

[bkim54]

I am creating an application where I want to create consumer groups, I understand if I have n partitions then the n+1 consumer will not receive any messages.

However I have noticed that during the rebalance some of the existing consumers end up with no partitions and the n+1 consumer group gets a partition.

In my application I don't want to stop the existing consumers from consuming, I would rather keep the existing consumers and not allow the n+1 consumer from joining the group.

I am trying to use Task<GroupInfo> tGroup = consumer.ListGroup(groupID, new TimeSpan(0, 0, 30)); but it does not seem to return the current number of members in the group until I start on the consumer, and by this time the n+1 consumer is already added to the group.

[ah-]

I think you might be able to create a consumer/producer without a group id first and use that to check.

[bkim54]

i cant seem to create a consumer without a groupID and the producer does not seem to retrieve the existing consumer group properly

[bkim54]

I worked around it by keeping track of how many partitions are currently assigned to a consumer and calling ListGroup within onPartitionsAssigned and checking how many members are in the group.

if there are more than n consumers I shut down the consumer that had no previous partitions.

this method works, but forces two extra consumer group rebalances (one because I have to add the n+1 consumer to the consumer group, and another because after i shut down the n+1 consumer the remaining n consumers have to rebalance)