Burrow not pulling consumers

[lmallonee]

Hi,

I have a cluster that only has zookeeper consumers, so I set up a kafka_zk consumer. When I curl to make sure everything is reading properly, the topics show up, but the consumers do not appear. In addition, I get the following error when I include the kafka consumer (it's currently commented out because of it):

msg":"failed to get partition count","type":"module","coordinator":"consumer","class":"kafka","name":"cars_production","topic":"__consumer_offsets","error":"kafka server: Request was for a topic or partition that does not exist on this broker."}

What could be making it so there are no consumers listed?

[lmallonee]

Here is my .toml file just in case: [general]
pidfile="/var/run/burrow/burrow.pid"
stdout-logfile=""
access-control-allow-origin="mysite.example.com"

[logging]
level="info"
filename="/var/log/burrow/burrow.log"
maxsize=100
maxbackups=10
maxage=10

[zookeeper]
servers=[ "zkserver:2181", "zkserver:2181", "zkserver:2181" ]
timeout=6
root-path="/burrow"

[client-profile.prod]
client-id="burrow-prod"
kafka-version="0.10.0"

[cluster.KAFKA_FE]
class-name="kafka"
servers=[ "kafkaserver:6667", "kafkaserver:6667", "kafkaserver:6667" ]
client-profile="prod"
topic-refresh=120
offset-refresh=30

[consumer.Kafka_FE]

class-name="kafka"

cluster="kafkacluster"

servers=[ "kafkaserver:6667", "kafkaserver:6667", "kafkaserver:6667" ]

client-profile="prod"

offsets-topic="__consumer_offsets"

start-latest=true

group-whitelist=".*"

group-blacklist=".*"

[consumer.KAFKA_ZK]
class-name="kafka_zk"
cluster="KAFKA_FE"
servers=[ "zkserver:2181", "zkserver:2181", "zkserver:2181" ]
group-whitelist=".*"

[httpserver.default]
address=":8002"

[storage.default]
class-name="inmemory"
workers=20
intervals=15
expire-group=604800
min-distance=1

~
~
"/etc/burrow/burrow.toml" 54L, 1576C

[zgseed]

the error msg says the topic __consumer_offsets does not exist in the kafka cluster you provide.
and the cluster field in the module consumer.Kafka_FE seems not configured in the cluster module. About the empty consumer list, make sure the consumer group has been executed in last 7 days.

[lmallonee]

Thank you, I didn't realize that I made a typo - that issue has been resolved.

As for the consumer list, that still remains an issue. This is being set up for prod, so the consumers are being executed, but it's not listing them out like it should.

[cluyihunter]

@lmallonee I ran into the same issue and fixed it in a weird way... try use all lower case characters in your consumer name

[lmallonee]

@cluyihunter I did do that, but there was no change - it's still spitting out:

{"level":"error","ts":1533070580.464172,"msg":"failed to get partition count","type":"module","coordinator":"consumer","class":"kafka","name":"kafkafe","topic":"__consumer_offsets","error":"kafka server: Request was for a topic or partition that does not exist on this broker."} {"level":"error","ts":1533070580.4642935,"msg":"failed to start consumer","type":"module","coordinator":"consumer","class":"kafka","name":"kafkafe","error":"kafka server: Request was for a topic or partition that does not exist on this broker."}

I'm thinking this is probably the root issue of why my consumers aren't showing up as well. For some reason it just skipped past reading the kafka consumer config and went to the zookeeper config until I changed it.

Just in case this helps, this is in prod, so __consumer_offsets is being updated frequently.

[lmallonee]

Quick update - still facing the same issue