search

Yolean / kubernetes-kafka

Kafka cluster as Kubernetes StatefulSet, plain manifests and config
Apache License 2.0
1.84k stars 733 forks source link

Burrow Exporter in CrashLoopBackOff when Kafka cluster has issues #284

Open amankedia opened 5 years ago

amankedia commented 5 years ago

I have referenced your Helm charts for Burrow and Burrow exporter and built Docker images for burrow exporter using https://github.com/jirwin/burrow_exporter/. However, when things are not in good shape with the Kafka cluster things go wrong with Burrow and Burrow exporter as well. The pod goes into CrashLoopBackOff state. Following is the log

time="2019-07-24T02:16:09Z" level=error msg="error listing clusters. Continuing." err="Get http://localhost:8000/v3/kafka: dial tcp 127.0.0.1:8000: connect: connection refused" time="2019-07-24T02:16:39Z" level=info msg="Scraping burrow..." timestamp=1563934599815846635 time="2019-07-24T02:16:39Z" level=error msg="error making request" endpoint="http://localhost:8000/v3/kafka" err="Get http://localhost:8000/v3/kafka: dial tcp 127.0.0.1:8000: connect: connection refused" time="2019-07-24T02:16:39Z" level=error msg="error retrieving cluster details" err="Get http://localhost:8000/v3/kafka: dial tcp 127.0.0.1:8000: connect: connection refused" time="2019-07-24T02:16:39Z" level=error msg="error listing clusters. Continuing." err="Get http://localhost:8000/v3/kafka: dial tcp 127.0.0.1:8000: connect: connection refused"

We do not have liveness and readiness probes defined for burrow exporter, how do we check it? My concern is any issue with the Kafka cluster should not put the burrow-exporter in CrashLoopBackOff state

solsson commented 5 years ago

So you're not using https://github.com/Yolean/kubernetes-kafka/tree/master/linkedin-burrow? Do you think that the crash loop can be avoided through configuration changes? Isn't it by burrow's design?

amankedia commented 5 years ago

@solsson I am using pretty much the same thing except the docker images which I have built using the Dockerfile at https://github.com/solsson/burrow_exporter/blob/master/Dockerfile

joyfulbean commented 2 years ago

I had the same error. Change the burrow config File to servers= [bootstrap:9092] instead of servers=["kafka-0.broker:9092", "kafka-1.broker:9092", "kafka-2.broker:9092"]. SVC is not broker so it will not work. In my case, after this change, it worked correctly.

[cluster.local]
class-name="kafka"
servers=[ "bootstrap:9092" ]
topic-refresh=60
offset-refresh=30

[consumer.local]
class-name="kafka"
cluster="local"
servers=[ "bootstrap:9092" ]
group-blacklist=""
group-whitelist=""