linkedin / Burrow

Kafka Consumer Lag Checking
Apache License 2.0
3.75k stars 798 forks source link

I get metrics for all of the topics but I don't see all of the consumer groups #583

Open avnerbarr opened 4 years ago

avnerbarr commented 4 years ago

I get metrics for all of the topics (i see all of the topic names in the metrics list), but for some reason not for all of the consumer groups.

gmcremon-mcafee commented 4 years ago

+1

I'm using: Burrow v1.2.2 Kafka 2.3.1 Zookeeper 3.5.6

When I hit the v3 API I can see my kafka cluster curl localhost:8000/v3/kafka/local { "error":false, "message":"cluster module detail returned", "module":{ "class-name":"kafka", "client-profile":{ "client-id":"burrow-lagchecker", "kafka-version":"0.8", "name":"", "sasl":null, "tls":null }, "offset-refresh":10, "servers":[ "kafka-kfk-0.dxlstreaming-kafka-headless.neweksdatabus.svc.cluster.local:9092", "kafka-kfk-1.dxlstreaming-kafka-headless.neweksdatabus.svc.cluster.local:9092" ], "topic-refresh":60 }, "request":{ "host":"kafka--0-0-8--monitoring-b47f6bbcb-kvb5v", "url":"/v3/kafka/local" } }

But, when i try to list the consumers it prints out an empty list and i do have some consumers running. curl localhost:8000/v3/kafka/local/consumer { "consumers": [], "error": false, "message": "consumer list returned", "request": { "host": "kafka--0-0-8--monitoring-b47f6bbcb-kvb5v", "url": "/v3/kafka/local/consumer" } }

I'm assuming @avnerbarr only sees info about topics in his metric exporter because burrow can see the brokers, but cannot see any consumers.

This started to happen when i migrated the kafka broker from 0.10.2.1 to 2.3.1

gmcremon-mcafee commented 4 years ago

Just saw there is no support for 2.3.0 or above yet #568.