Error Init Kafka Client - client has run out of available brokers to talk to (Is your cluster reachable?)

[leoberbert]

When trying to start the exporter for kafka, it is showing an error as shown below:

./kafka_exporter --zookeeper.server=myserver_zookeeper:2181 --log.enable-sarama INFO[0000] Starting kafka_exporter (version=, branch=, revision=) source="kafka_exporter.go:535" INFO[0000] Build context (go=go1.16.4, user=, date=) source="kafka_exporter.go:536" [sarama] 2021/08/27 13:33:28 Initializing new client [sarama] 2021/08/27 13:33:28 client/metadata fetching metadata for all topics from broker kafka:9092 [sarama] 2021/08/27 13:33:28 Failed to connect to broker kafka:9092: dial tcp: lookup kafka on 10.112.32.23:53: server misbehaving [sarama] 2021/08/27 13:33:28 client/metadata got error from broker -1 while fetching metadata: dial tcp: lookup kafka on 10.112.32.23:53: server misbehaving [sarama] 2021/08/27 13:33:28 client/metadata no available broker to send metadata request to [sarama] 2021/08/27 13:33:28 client/brokers resurrecting 1 dead seed brokers [sarama] 2021/08/27 13:33:28 client/metadata retrying after 250ms... (3 attempts remaining) [sarama] 2021/08/27 13:33:28 client/metadata fetching metadata for all topics from broker kafka:9092 [sarama] 2021/08/27 13:33:29 Failed to connect to broker kafka:9092: dial tcp: lookup kafka on 10.112.32.23:53: server misbehaving [sarama] 2021/08/27 13:33:29 client/metadata got error from broker -1 while fetching metadata: dial tcp: lookup kafka on 10.112.32.23:53: server misbehaving [sarama] 2021/08/27 13:33:29 client/metadata no available broker to send metadata request to [sarama] 2021/08/27 13:33:29 client/brokers resurrecting 1 dead seed brokers [sarama] 2021/08/27 13:33:29 client/metadata retrying after 250ms... (2 attempts remaining) [sarama] 2021/08/27 13:33:29 client/metadata fetching metadata for all topics from broker kafka:9092 [sarama] 2021/08/27 13:33:29 Failed to connect to broker kafka:9092: dial tcp: lookup kafka on 10.112.32.23:53: server misbehaving [sarama] 2021/08/27 13:33:29 client/metadata got error from broker -1 while fetching metadata: dial tcp: lookup kafka on 10.112.32.23:53: server misbehaving [sarama] 2021/08/27 13:33:29 client/metadata no available broker to send metadata request to [sarama] 2021/08/27 13:33:29 client/brokers resurrecting 1 dead seed brokers [sarama] 2021/08/27 13:33:29 client/metadata retrying after 250ms... (1 attempts remaining) [sarama] 2021/08/27 13:33:29 client/metadata fetching metadata for all topics from broker kafka:9092 [sarama] 2021/08/27 13:33:29 Failed to connect to broker kafka:9092: dial tcp: lookup kafka on 10.112.32.23:53: server misbehaving [sarama] 2021/08/27 13:33:29 client/metadata got error from broker -1 while fetching metadata: dial tcp: lookup kafka on 10.112.32.23:53: server misbehaving [sarama] 2021/08/27 13:33:29 client/metadata no available broker to send metadata request to [sarama] 2021/08/27 13:33:29 client/brokers resurrecting 1 dead seed brokers [sarama] 2021/08/27 13:33:29 Closing Client FATA[0000] Error Init Kafka Client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?) source="kafka_exporter.go:643"

I've tried other ways and it doesn't collect. If I run the kafka command itself to list the metrics, it returns normally.

./kafka-topics.sh --zookeeper myserver_zookeeper:2181 --list

queuing.error queuing.memory queuing.cpu

[rbosneag]

same issue here as @leoberbert

how did you solve it?

[Emma-uestc]

I also encountered this issue when I use 1.4.2,my kafka is 2.11-0.11.0.0, I changed the kafka exporter tho version 1.2 and it worked,but it does not support --zookeeper.server.But it said "Support Apache Kafka version 0.10.1.0(and later) " in readme,maybe it do not update the docs

[jiangyunpeng]

+1

[gauravdigilytics]

+1

I0919 17:32:21.329393 1 kafka_exporter.go:823] Starting kafka_exporter (version=1.8.0, branch=master, revision=477643dd764c46ddd804ac4bd4022be877e3085b) F0919 17:32:22.193214 1 kafka_exporter.go:924] Error Init Kafka Client: kafka: client has run out of available brokers to talk to: dial tcp: lookup kafka on 10.0.0.10:53: no such host

Seems like it is looking for host "kafka"...however my broker addresses are like kaf1, kaf2 etc, and I have verified they are properly configured as well as accessible.

[leoberbert]

Dear @gauravdigilytics

In my case I was using Kerberos and I resolved it as follows.

nohup ./kafka_exporter \
  --kafka.server=srv01.interno.work.com.br:6667 \
  --kafka.server=srv02.interno.work.com.br:6667 \
  --kafka.server=srv03.interno.work.com.br:6667 \
  --kafka.server=srv04.interno.work.com.br:6667 \
  --kafka.version=2.0.0 \
  --sasl.enabled \
  --sasl.mechanism=GSSAPI \
  --sasl.service-name=kafka \
  --sasl.kerberos-config-path=/etc/krb5.conf \
  --sasl.realm=FD-FLOW\
  --sasl.keytab-path=/etc/security/keytabs/kafka.service.keytab \
  --sasl.kerberos-auth-type=keytabAuth \
  --sasl.username=kafka/srv01.interno.work.com.br \
  --web.listen-address=:9308 \
  --web.telemetry-path=/metrics \
  --topic.filter=.* \
  --group.filter=.* \
  --offset.show-all \
  --log.level=info > kafka_exporter.log 2>&1 &

Hope this helps,

Regards,

Leonardo Berbert

[gauravdigilytics]

@leoberbert Thanks for the quick response.

Previously, I was attempting to install it via YAML, but I may have overlooked something. I've now successfully installed it using Helm. For anyone else following along, here’s the Helm command I used with the prometheus-kafka-exporter chart: helm install kafka-metrics prometheus-community/prometheus-kafka-exporter --namespace monitoring --set kafkaServer="{kaf1.svc.cluster.local:9092,kaf2.svc.cluster.local:9092,kaf3.svc.cluster.local:9092}"

Make sure to update the helm release-name as per your choice and Kafka broker addresses according to your setup!

Moreover, to install Prometheus server, use prometheus chart: helm install kafka-metrics-prometheus prometheus-community/prometheus --namespace monitoring