search

Telefonica / prometheus-kafka-adapter

Use Kafka as a remote storage database for Prometheus (remote write only)
Apache License 2.0
364 stars 135 forks source link

unable to see any metrics in kafka topic help in debuging #85

Closed rambabubolla closed 2 years ago

rambabubolla commented 2 years ago

Unable see metrics in kafka topic and also unable to debug

able to see logs in promothisys

level=info ts=2022-01-28T03:58:11.350Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/config_out/prometheus.env.yaml ts=2022-01-28T03:58:11.360Z caller=dedupe.go:112 component=remote level=info remote_name=8a1db9 url=http://prometheus-kafka-adapter:8080/receive msg="Starting WAL watcher" queue=8a1db9 ts=2022-01-28T03:58:11.361Z caller=dedupe.go:112 component=remote level=info remote_name=8a1db9 url=http://prometheus-kafka-adapter:8080/receive msg="Replaying WAL" queue=8a1db9 level=info ts=2022-01-28T03:58:11.368Z caller=kubernetes.go:253 component="discovery manager scrape" discovery=k8s msg="Using pod service account via in-cluster config"

unable see any POST method all the logs having only GET method

{"fields.time":"2022-01-28T06:48:58Z","ip":"10.44.0.1","latency":3876226,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-28T06:48:58Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-01-28T06:49:06Z","ip":"10.40.1.106","latency":1865536,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-28T06:49:06Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-01-28T06:49:07Z","ip":"10.44.0.1","latency":3651721,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-28T06:49:07Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-01-28T06:49:08Z","ip":"10.44.0.1","latency":3860686,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-28T06:49:08Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-01-28T06:49:10Z","ip":"10.39.0.175","latency":3794815,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-28T06:49:10Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-01-28T06:49:11Z","ip":"10.40.1.105","latency":4061371,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-28T06:49:11Z","user-agent":"

rambabubolla commented 2 years ago

prometheusSpec:

  remoteWrite:
    - url: "http://prometheus-kafka-adapter:8080/receive"
rambabubolla commented 2 years ago

kubectl logs prometheus-kafka-adapter-59bdfb49dd-4knsh {"level":"info","msg":"creating kafka producer","time":"2022-02-01T05:44:54Z"} [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

[GIN-debug] GET /metrics --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers) [GIN-debug] POST /receive --> main.receiveHandler.func1 (3 handlers) [GIN-debug] Environment variable PORT="8080" [GIN-debug] Listening and serving HTTP on :8080 {"fields.time":"2022-02-01T05:44:57Z","ip":"10.44.0.1","latency":2450050,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T05:44:57Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-02-01T05:44:57Z","ip":"10.44.0.1","latency":2534833,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T05:44:57Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-02-01T05:45:03Z","ip":"10.40.1.105","latency":4347348,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T05:45:03Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-02-01T05:45:03Z","ip":"10.39.0.175","latency":4598425,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T05:45:03Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-02-01T05:45:05Z","ip":"10.40.1.106","latency":4551463,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T05:45:05Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-02-01T05:45:07Z","ip":"10.44.0.1","latency":4517302,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T05:45:07Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-02-01T05:45:07Z","ip":"10.44.0.1","latency":4392330,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T05:45:07Z","user-agent":"kube-probe/1.19"}

rambabubolla commented 2 years ago

able to verify with kafka broker is working fine kafkacat -L -b cluster-kafka-0.cluster-kafka-brokers.sma.svc.cluster.local |grep test topic "metric-test" with 4 partitions: / #

rambabubolla commented 2 years ago

tried with image version 1.8.0 tag but same result there is no POST call kubectl logs prometheus-kafka-adapter-8457c4c8f7-mgvsr {"level":"info","msg":"creating kafka producer","time":"2022-02-01T10:53:07Z"} [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

[GIN-debug] GET /metrics --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers) [GIN-debug] GET /healthz --> main.main.func1 (3 handlers) [GIN-debug] POST /receive --> main.receiveHandler.func1 (3 handlers) [GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value. Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details. [GIN-debug] Environment variable PORT="8080" [GIN-debug] Listening and serving HTTP on :8080 {"fields.time":"2022-02-01T10:53:08Z","ip":"10.44.0.1","latency":117992,"level":"info","method":"GET","msg":"","path":"/healthz","status":200,"time":"2022-02-01T10:53:08Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-02-01T10:53:14Z","ip":"10.40.1.106","latency":2443091,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T10:53:14Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-02-01T10:53:14Z","ip":"10.44.0.1","latency":62471,"level":"info","method":"GET","msg":"","path":"/healthz","status":200,"time":"2022-02-01T10:53:14Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-02-01T10:53:15Z","ip":"10.39.0.175","latency":3589790,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T10:53:15Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-02-01T10:53:18Z","ip":"10.44.0.1","latency":52751,"level":"info","method":"GET","msg":"","path":"/healthz","status":200,"time":"2022-02-01T10:53:18Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-02-01T10:53:22Z","ip":"10.40.1.105","latency":4293557,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-01T10:53:22Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-02-01T10:53:24Z","ip":"10.44.0.1","latency":50001,"level":"info","method":"GET","msg":"","path":"/healthz","status":200,"time":"2022-02-01T10:53:24Z","user-agent":"kube

rambabubolla commented 2 years ago

ubectl logs prometheus-kafka-adapter-698697ffd7-ldglr {"level":"info","msg":"creating kafka producer","time":"2022-02-02T10:46:19Z"} [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

[GIN-debug] GET /metrics --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers) [GIN-debug] GET /healthz --> main.main.func1 (3 handlers) [GIN-debug] POST /receive --> main.receiveHandler.func1 (3 handlers) [GIN-debug] [WARNING] You trusted all proxies, this is NOT safe. We recommend you to set a value. Please check https://pkg.go.dev/github.com/gin-gonic/gin#readme-don-t-trust-all-proxies for details. [GIN-debug] Environment variable PORT="8080" [GIN-debug] Listening and serving HTTP on :8080 {"fields.time":"2022-02-02T10:46:21Z","ip":"10.44.0.1","latency":121703,"level":"info","method":"GET","msg":"","path":"/healthz","status":200,"time":"2022-02-02T10:46:21Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-02-02T10:46:22Z","ip":"10.39.0.175","latency":2223745,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-02T10:46:22Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-02-02T10:46:28Z","ip":"10.40.1.106","latency":3907646,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-02T10:46:28Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-02-02T10:46:29Z","ip":"10.44.0.1","latency":58601,"level":"info","method":"GET","msg":"","path":"/healthz","status":200,"time":"2022-02-02T10:46:29Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-02-02T10:46:31Z","ip":"10.40.1.105","latency":3795813,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-02T10:46:31Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-02-02T10:46:31Z","ip":"10.44.0.1","latency":51091,"level":"info","method":"GET","msg":"","path":"/healthz","status":200,"time":"2022-02-02T10:46:31Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-02-02T10:46:37Z","ip":"10.39.0.175","latency":4026499,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-02-02T10:46:37Z","user-agent"

rambabubolla commented 2 years ago

working fine with ipaddress instead of prometheus-kafka-adapte

remoteWrite: