search

mailgun / kafka-pixy

gRPC/REST proxy for Kafka
Apache License 2.0
774 stars 118 forks source link

Refactor subscriber #121

Closed horkhe closed 7 years ago

horkhe commented 7 years ago

Subscriber used to set watches only on the change in the number of consumer group members. To detect changes in member subscriptions (list of topics they consume) a trick had been used to remove registration and then add it again with an updated topic list. To avoid unnecessary rebalancing on such temporary deregistrations an internal RebalancingDelay timeout was used to postpone rebalancing in expectation that registration may follow.

In this PR we vendored a mailgun branch of kazoo library that allows setting watches on member subscriptions so that deregistration to update a list of topics became unnecessary. As a result the logic became simpler.

Besides a several unnecessary optimizations were removed and one of them was a cause of #120.

thrawn01 commented 7 years ago

Alot of this is over my head, because I don't have a full running theory of how kafka-pixy works. At some point in the future I would love to corner you and ask a bunch of questions. =)