search

haskell-works / hw-kafka-client

Kafka client for Haskell, including auto-rebalancing consumers
MIT License
140 stars 50 forks source link

Questions around rebalance callback #106

Closed alexbiehl closed 5 years ago

alexbiehl commented 5 years ago

We found that in newConsumer we set a rebalance callback:

https://github.com/haskell-works/hw-kafka-client/blob/2d33327f94802048aeb6b5818db381edca544db7/src/Kafka/Consumer.hs#L117

Internally this is happening through librdkafkas rd_kafka_conf_set_rebalance_cb.

Now, we found that librdkafka documentation for rd_kafka_conf_set_rebalance_cb mentions 

 * Registering a \p rebalance_cb turns off librdkafka's automatic
 * partition assignment/revocation and instead delegates that responsibility
 * to the application's \p rebalance_cb.

(see https://github.com/edenhill/librdkafka/blob/4fb7c9924c14f44c4f1490b80b8bd714098dd90b/src/rdkafka.h#L1593-L1666)

It seems by setting the rebalance callback internally we deactivate librdkafkas assignment/revocation strategy and the callback we set return () doesn't handle the cases at all!

What are we missing here? Is this an oversight or is there another corner we need to look at?

AlexeyRaga commented 5 years ago

Yes, you are right, we set the callback and the callback assumes the responsibility for assignment/revocation. And it does, here: https://github.com/haskell-works/hw-kafka-client/blob/2d33327f94802048aeb6b5818db381edca544db7/src/Kafka/Consumer/Callbacks.hs#L71

The callback that is provided by a user (the one that returns ()) is being wrapped with setRebalanceCallback which does the job.

The "user-level" rebalance callback is purely informational and it does not interfere with the assignment/revocation logic.

AlexeyRaga commented 5 years ago

@alexbiehl Have I answered your question? If so, can we close this issue?

alexbiehl commented 5 years ago

Oh yes, thank you! @shlevy is following up on this.

On Thu, Sep 5, 2019, 19:27 Alexey Raga notifications@github.com wrote:

@alexbiehl https://github.com/alexbiehl Have I answered your question? If so, can we close this issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/haskell-works/hw-kafka-client/issues/106?email_source=notifications&email_token=AAOKFCM5SN4GNYMCID44TF3QIE6OPA5CNFSM4INW2R3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6ABDZY#issuecomment-528486887, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOKFCMZX3OVMXK53YY7VP3QIE6OPANCNFSM4INW2R3A .