search

danielqsj / kafka_exporter

Kafka exporter for Prometheus
Apache License 2.0
2.19k stars 610 forks source link

Error: "Cannot get offset of group kafka: broker not connected" source="kafka_exporter.go:384" #198

Open hemalsolgama opened 3 years ago

hemalsolgama commented 3 years ago

Hello,

We've configured kafka-exporter with prometheus in GKE environment. On development environment, it has errors but prometheus returns accurate metrics result. However, in production environment, we don't see accurate data.

Say for example, we've set monitoring in Grafana to check kafka_consumergroup_lag_sum value but this data is incomplete. Prometheus only shows one or two partitions' value and for the rest of the partitions, it shows "-1".

While checking the logs of kafka-exporter pod, we came to know that it's full of this kind of errors: level=error msg="Cannot get offset of group : kafka: broker not connected" source="kafka_exporter.go:384"

Kafka brokers are connected properly but we couldn't figure out why prometheus doesn't show all data.

OuesFa commented 3 years ago

Same issue on our side ! Using Kafka Exporter with the Strimzi Kafka operator in GKE. Plenty of logs showing 

time="2021-01-08T11:35:59Z" level=error msg="Cannot get offset of group my-group: kafka: broker not connected" source="kafka_exporter.go:384"
time="2021-01-08T11:35:59Z" level=error msg="Cannot get offset of group my-group: kafka: broker not connected" source="kafka_exporter.go:384"

But the Kafka cluster seems to be up & running well.

@hemalsolgama did you find the issue please ?

hemalsolgama commented 3 years ago

@OuesFa Nope.. I'm also trying to find a solution for this because we have incomplete data in prometheus due to this..

senthil13 commented 3 years ago

+1 I too facing the same issue here. But it was collecting metric even though it shows broker disconnects in the logs. With the same config DEV is working fine for me, Only difference is dev has 74k metrics and pilot has 134697 metrics. May be its unable to hold that much load.

u605902 commented 3 years ago

I am getting the same error Cannot get offset of group : kafka: broker not connected" source="kafka_exporter.go:412"

we did use the image v1.3.1 and v1.3.0. both and I am seeing the above error. it's connecting to the confluent version of kafka 6.1.0

any solution is greatly appreciated.

Joseph94m commented 3 years ago

Same issue. It's weird because we use the same version on 2 different clusters. One of them shows the issue, and the other doesnt

sdandamudi5c commented 2 years ago

Was anyone able to find a solution to this issue?