search

confluentinc / librdkafka

The Apache Kafka C/C++ library
Other
204 stars 3.14k forks source link

why it don't call Rebalance after rd_kafka_consumer_poll #1810

Closed heketong closed 6 years ago

heketong commented 6 years ago

Read the FAQ first: https://github.com/edenhill/librdkafka/wiki/FAQ

Description

it don't call Rebalance after rd_kafka_consumer_poll,so i can't consume a thing from one topic. the librdkafka debug logs are below. This question has been bothering me for a long time,please help me out,tks!

How to reproduce

<your steps how to reproduce goes here, or remove section if not relevant>

IMPORTANT: Always try to reproduce the issue on the latest released version (see https://github.com/edenhill/librdkafka/releases), if it can't be reproduced on the latest version the issue has been fixed.

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

Topic config

request.required.acks = 1 request.timeout.ms = 5000 message.timeout.ms = 300000 queuing.strategy = fifo produce.offset.report = false partitioner = consistent_random compression.codec = inherit auto.commit.enable = false auto.commit.interval.ms = 60 auto.offset.reset = smallest offset.store.path = . offset.store.sync.interval.ms = -1 offset.store.method = broker consume.callback.max.messages = 0

edenhill commented 6 years ago

In worst case you may need to wait for up to session.timeout.ms (def 30 seconds) to get an assignment. Your debug output only covers a couple of seconds.

Please also share your (relevant) code.

heketong commented 6 years ago

add something: i got time out(timeout:one seconds) when i call rd_kafka_consumer_poll. and in my case it consume two topic in cycle,when one ie EOF,it will consume another one. the first one always be ok(call rebalance after rd_kafka_consumer_poll), but the second always be not ok(do not call rebalance after rd_kafka_consumer_poll)

heketong commented 6 years ago

but why i need to wait for up to session.timeout.ms(def 30 seconds)? you mean librdkafka lost the connect with kafka broker?

edenhill commented 6 years ago

If the group exists and has had a member recently fall out it may take up to session.timeout.ms to rebalance the group.

heketong commented 6 years ago

but when i start two client to consume one topic(with 2 partition),when i kill one client, the other client's log shows me that it will rebalance the group immediately. why?

edenhill commented 6 years ago

If a client is shut down in a controlled fashion it will send a LeaveGroup request which immediately triggers a rebalance.