search

danielqsj / kafka_exporter

Kafka exporter for Prometheus
Apache License 2.0
2.22k stars 615 forks source link

kafka_exporterv1.3.1 running failed by kafka 0.10.2 #236

Open oldthreefeng opened 3 years ago

oldthreefeng commented 3 years ago

kafka version is 0.10.2 . (Support Apache Kafka version 0.10.1.0 (and later).)

my yaml

# Source: kafka-exporter/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kafka-exporter
  labels:
    app.kubernetes.io/name: kafka-exporter
    helm.sh/chart: kafka-exporter-1.0.0
    app.kubernetes.io/instance: kafka-exporter
    app.kubernetes.io/managed-by: Tiller
  namespace: monitor
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: kafka-exporter
      app.kubernetes.io/instance: kafka-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/name: kafka-exporter
        app.kubernetes.io/instance: kafka-exporter
    spec:
      containers:
        - name: kafka-exporter
          image: "danielqsj/kafka-exporter:v1.3.1"
          imagePullPolicy: IfNotPresent
          args:
            - "--kafka.server=192.168.30.5:9092"
            - "--kafka.server=192.168.30.6:9092"
            - "--kafka.server=192.168.30.7:9092"
            - --log.level=info
            - --log.enable-sarama
          ports:
            - name: metrics
              containerPort: 9308
              protocol: TCP
          resources:
            limits:
              cpu: 500m
            requests:
              cpu: 500m

          livenessProbe:
            httpGet:
              path: /health
              port: 9308
            initialDelaySeconds: 30
            timeoutSeconds: 10

when running in cluster. failed? 

k logs -n monitor kafka-exporter-75b95fccfb-b2tvf 
time="2021-06-24T08:40:32Z" level=info msg="Starting kafka_exporter (version=, branch=, revision=)" source="kafka_exporter.go:535"
[sarama] 2021/06/24 08:40:32 Initializing new client
[sarama] 2021/06/24 08:40:32 client/metadata fetching metadata for all topics from broker 192.168.30.7:9092
time="2021-06-24T08:40:32Z" level=info msg="Build context (go=go1.16, user=, date=)" source="kafka_exporter.go:536"
[sarama] 2021/06/24 08:40:32 Connected to broker at 192.168.30.7:9092 (unregistered)
[sarama] 2021/06/24 08:40:32 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:32 Closed connection to broker 192.168.30.7:9092
[sarama] 2021/06/24 08:40:32 client/metadata fetching metadata for all topics from broker 192.168.30.6:9092
[sarama] 2021/06/24 08:40:32 Connected to broker at 192.168.30.6:9092 (unregistered)
[sarama] 2021/06/24 08:40:32 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:32 Closed connection to broker 192.168.30.6:9092
[sarama] 2021/06/24 08:40:32 client/metadata fetching metadata for all topics from broker 192.168.30.5:9092
[sarama] 2021/06/24 08:40:32 Connected to broker at 192.168.30.5:9092 (unregistered)
[sarama] 2021/06/24 08:40:32 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:32 Closed connection to broker 192.168.30.5:9092
[sarama] 2021/06/24 08:40:32 client/metadata no available broker to send metadata request to
[sarama] 2021/06/24 08:40:32 client/brokers resurrecting 3 dead seed brokers
[sarama] 2021/06/24 08:40:32 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama] 2021/06/24 08:40:32 client/metadata fetching metadata for all topics from broker 192.168.30.7:9092
[sarama] 2021/06/24 08:40:32 Connected to broker at 192.168.30.7:9092 (unregistered)
[sarama] 2021/06/24 08:40:32 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:32 Closed connection to broker 192.168.30.7:9092
[sarama] 2021/06/24 08:40:32 client/metadata fetching metadata for all topics from broker 192.168.30.6:9092
[sarama] 2021/06/24 08:40:32 Connected to broker at 192.168.30.6:9092 (unregistered)
[sarama] 2021/06/24 08:40:32 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:32 Closed connection to broker 192.168.30.6:9092
[sarama] 2021/06/24 08:40:32 client/metadata fetching metadata for all topics from broker 192.168.30.5:9092
[sarama] 2021/06/24 08:40:32 Connected to broker at 192.168.30.5:9092 (unregistered)
[sarama] 2021/06/24 08:40:32 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:32 Closed connection to broker 192.168.30.5:9092
[sarama] 2021/06/24 08:40:32 client/metadata no available broker to send metadata request to
[sarama] 2021/06/24 08:40:32 client/brokers resurrecting 3 dead seed brokers
[sarama] 2021/06/24 08:40:32 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama] 2021/06/24 08:40:33 client/metadata fetching metadata for all topics from broker 192.168.30.7:9092
[sarama] 2021/06/24 08:40:33 Connected to broker at 192.168.30.7:9092 (unregistered)
[sarama] 2021/06/24 08:40:33 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:33 Closed connection to broker 192.168.30.7:9092
[sarama] 2021/06/24 08:40:33 client/metadata fetching metadata for all topics from broker 192.168.30.6:9092
[sarama] 2021/06/24 08:40:33 Connected to broker at 192.168.30.6:9092 (unregistered)
[sarama] 2021/06/24 08:40:33 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:33 Closed connection to broker 192.168.30.6:9092
[sarama] 2021/06/24 08:40:33 client/metadata fetching metadata for all topics from broker 192.168.30.5:9092
[sarama] 2021/06/24 08:40:33 Connected to broker at 192.168.30.5:9092 (unregistered)
[sarama] 2021/06/24 08:40:33 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:33 Closed connection to broker 192.168.30.5:9092
[sarama] 2021/06/24 08:40:33 client/metadata no available broker to send metadata request to
[sarama] 2021/06/24 08:40:33 client/brokers resurrecting 3 dead seed brokers
[sarama] 2021/06/24 08:40:33 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama] 2021/06/24 08:40:33 client/metadata fetching metadata for all topics from broker 192.168.30.7:9092
[sarama] 2021/06/24 08:40:33 Connected to broker at 192.168.30.7:9092 (unregistered)
[sarama] 2021/06/24 08:40:33 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:33 Closed connection to broker 192.168.30.7:9092
[sarama] 2021/06/24 08:40:33 client/metadata fetching metadata for all topics from broker 192.168.30.6:9092
[sarama] 2021/06/24 08:40:33 Connected to broker at 192.168.30.6:9092 (unregistered)
[sarama] 2021/06/24 08:40:33 client/metadata got error from broker -1 while fetching metadata: EOF
[sarama] 2021/06/24 08:40:33 Closed connection to broker 192.168.30.6:9092
[sarama] 2021/06/24 08:40:33 client/metadata fetching metadata for all topics from broker 192.168.30.5:9092
[sarama] 2021/06/24 08:40:33 Connected to broker at 192.168.30.5:9092 (unregistered)
[sarama] 2021/06/24 08:40:33 client/metadata got error from broker -1 while fetching metadata: EOF
time="2021-06-24T08:40:33Z" level=fatal msg="Error Init Kafka Client: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)" source="kafka_exporter.go:643"
[sarama] 2021/06/24 08:40:33 Closed connection to broker 192.168.30.5:9092
[sarama] 2021/06/24 08:40:33 client/metadata no available broker to send metadata request to
[sarama] 2021/06/24 08:40:33 client/brokers resurrecting 3 dead seed brokers
[sarama] 2021/06/24 08:40:33 Closing Client
oldthreefeng commented 3 years ago

like #224 . fix image to 1.2.0 . connect success

 k logs -n monitor kafka-exporter-5bf54fbbfb-zq64z 
[sarama] 2021/06/24 08:49:11 Initializing new client
time="2021-06-24T08:49:11Z" level=info msg="Starting kafka_exporter (version=1.2.0, branch=HEAD, revision=830660212e6c109e69dcb1cb58f5159fe3b38903)" source="kafka_exporter.go:474"
time="2021-06-24T08:49:11Z" level=info msg="Build context (go=go1.9, user=travis@travis-job-2baddb7a-390a-490a-9723-d39790ff6c41, date=20180707-14:33:44)" source="kafka_exporter.go:475"
[sarama] 2021/06/24 08:49:11 client/metadata fetching metadata for all topics from broker 192.168.30.6:9092
[sarama] 2021/06/24 08:49:11 Connected to broker at 192.168.30.6:9092 (unregistered)
[sarama] 2021/06/24 08:49:11 client/brokers registered new broker #101 at 192.168.30.5:9092
[sarama] 2021/06/24 08:49:11 client/brokers registered new broker #103 at 192.168.30.7:9092
[sarama] 2021/06/24 08:49:11 client/brokers registered new broker #102 at 192.168.30.6:9092
[sarama] 2021/06/24 08:49:11 client/metadata found some partitions to be leaderless
[sarama] 2021/06/24 08:49:11 client/metadata retrying after 250ms... (3 attempts remaining)
[sarama] 2021/06/24 08:49:11 client/metadata fetching metadata for all topics from broker 192.168.30.6:9092
[sarama] 2021/06/24 08:49:11 client/metadata found some partitions to be leaderless
[sarama] 2021/06/24 08:49:11 client/metadata retrying after 250ms... (2 attempts remaining)
[sarama] 2021/06/24 08:49:11 client/metadata fetching metadata for all topics from broker 192.168.30.6:9092
[sarama] 2021/06/24 08:49:11 client/metadata found some partitions to be leaderless
[sarama] 2021/06/24 08:49:11 client/metadata retrying after 250ms... (1 attempts remaining)
[sarama] 2021/06/24 08:49:12 client/metadata fetching metadata for all topics from broker 192.168.30.6:9092
[sarama] 2021/06/24 08:49:12 client/metadata found some partitions to be leaderless
[sarama] 2021/06/24 08:49:12 Successfully initialized new client
time="2021-06-24T08:49:12Z" level=info msg="Done Init Clients" source="kafka_exporter.go:213"
time="2021-06-24T08:49:12Z" level=info msg="Listening on :9308" source="kafka_exporter.go:499"

when trying to get metrics curl clusterIP:9308/metrcis it hangs. and the pod log like this.

 k logs -n monitor kafka-exporter-56bc8697b7-n52pr
time="2021-06-24T08:53:17Z" level=info msg="Starting kafka_exporter (version=1.2.0, branch=HEAD, revision=830660212e6c109e69dcb1cb58f5159fe3b38903)" source="kafka_exporter.go:474"
time="2021-06-24T08:53:17Z" level=info msg="Build context (go=go1.9, user=travis@travis-job-2baddb7a-390a-490a-9723-d39790ff6c41, date=20180707-14:33:44)" source="kafka_exporter.go:475"
time="2021-06-24T08:53:18Z" level=info msg="Done Init Clients" source="kafka_exporter.go:213"
time="2021-06-24T08:53:18Z" level=info msg="Listening on :9308" source="kafka_exporter.go:499"
time="2021-06-24T08:53:48Z" level=error msg="Cannot get leader of topic springcloud_config_t1 partition 8: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes." source="kafka_exporter.go:275"
time="2021-06-24T08:53:50Z" level=error msg="Cannot get current offset of topic springcloud_config_t1 partition 8: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes." source="kafka_exporter.go:284"
time="2021-06-24T08:53:53Z" level=error msg="Cannot get oldest offset of topic springcloud_config_t1 partition 8: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes." source="kafka_exporter.go:296"
time="2021-06-24T08:53:53Z" level=error msg="Cannot get leader of topic springcloud_config_t1 partition 9: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes." source="kafka_exporter.go:275"
time="2021-06-24T08:53:56Z" level=error msg="Cannot get current offset of topic springcloud_config_t1 partition 9: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes." source="kafka_exporter.go:284"
time="2021-06-24T08:53:58Z" level=error msg="Cannot get oldest offset of topic springcloud_config_t1 partition 9: kafka server: In the middle of a leadership election, there is currently no leader for this partition and hence it is unavailable for writes." source="kafka_exporter.go:296"