search

linkedin / Burrow

Kafka Consumer Lag Checking
Apache License 2.0
3.72k stars 796 forks source link

Burrow does not detect consumer groups #370

Open lgabo opened 6 years ago

lgabo commented 6 years ago

We have a nightly run where we (in the following order)

  1. install and run kafka 1.0.0
  2. install and run burrow
  3. creating topics in kafka then we run our apps (and some tests) which create consumer groups which commit to kafka. The topic offsets and consumer group lags are periodically sent from burrow to grafana. In the morning when I check grafana it shows no info about consumer groups (the topic offsets are available). When I ask burrow manually via rest api for consumer groups it returns an empty list, although kafka's kafka-consumer-groups.sh script returns all available consumer groups (and lags) created by our apps. After restarting burrow it starts to return consumer groups too (not only topics) which appear in grafana then. What could be the problem here? Should we install and start burrow as last step in our nightly run?
bkochendorfer commented 6 years ago

I am seeing this same issue. Is there some configuration we can tweak for refreshing consumer list?

toddpalino commented 6 years ago

The consumer lists should update automatically as long as the consumers are committing offsets. Are they committing to Zookeeper or to Kafka?

sumitkumarm commented 5 years ago

Is this still the default Burrow behavior? What if Burrow restarts and some of the consumer groups do not have any offsets being committed for some time? Will those consumer groups show up as "not found" till they start committing offsets?

@toddpalino

6opuc commented 5 years ago

@toddpalino I'm facing the same issue(in similar use case as @lgabo ) and last messages in burrow logs are:

  1. ...Recv loop terminated...
  2. ...Send loop terminated...

I've searched for "Recv loop terminated" in source code and found that these messages belongs to https://github.com/samuel/go-zookeeper. I've even found possible cause of this issue: https://github.com/samuel/go-zookeeper/issues/133. Do you think that these issues are related?