Liveness probe and Readiness probe failed with 404

[rambabubolla]

ncn-m001:~ # kubectl logs prometheus-kafka-adapter-857659c4d7-mqr2d {"level":"warning","msg":"invalid serialization format, using json","serialization-format-value":"","time":"2022-01-04T14:51:24Z"} {"level":"info","msg":"creating kafka producer","time":"2022-01-04T14:51:24Z"} [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.

• using env: export GIN_MODE=release
• using code: gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET /metrics --> github.com/gin-gonic/gin.WrapH.func1 (3 handlers) [GIN-debug] POST /receive --> main.receiveHandler.func1 (3 handlers) [GIN-debug] Environment variable PORT="8080" [GIN-debug] Listening and serving HTTP on :8080 {"fields.time":"2022-01-04T14:51:24Z","ip":"10.46.1.48","latency":5030273,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-04T14:51:24Z","user-agent":"Prometheus/2.18.1"} {"fields.time":"2022-01-04T14:51:28Z","ip":"10.46.0.0","latency":650,"level":"info","method":"GET","msg":"","path":"/healthz","status":404,"time":"2022-01-04T14:51:28Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-01-04T14:51:34Z","ip":"10.46.0.59","latency":4603921,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-04T14:51:34Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-01-04T14:51:34Z","ip":"10.46.0.0","latency":969,"level":"info","method":"GET","msg":"","path":"/healthz","status":404,"time":"2022-01-04T14:51:34Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-01-04T14:51:34Z","ip":"10.41.0.49","latency":4658771,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-04T14:51:34Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-01-04T14:51:38Z","ip":"10.46.0.0","latency":930,"level":"info","method":"GET","msg":"","path":"/healthz","status":404,"time":"2022-01-04T14:51:38Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-01-04T14:51:38Z","ip":"10.41.0.163","latency":4689842,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-04T14:51:38Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-01-04T14:51:44Z","ip":"10.46.0.0","latency":790,"level":"info","method":"GET","msg":"","path":"/healthz","status":404,"time":"2022-01-04T14:51:44Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-01-04T14:51:48Z","ip":"10.46.0.0","latency":770,"level":"info","method":"GET","msg":"","path":"/healthz","status":404,"time":"2022-01-04T14:51:48Z","user-agent":"kube-probe/1.19"} {"fields.time":"2022-01-04T14:51:49Z","ip":"10.46.0.59","latency":4589961,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-04T14:51:49Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-01-04T14:51:49Z","ip":"10.41.0.49","latency":2449216,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-04T14:51:49Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-01-04T14:51:53Z","ip":"10.41.0.163","latency":4463612,"level":"info","method":"GET","msg":"","path":"/metrics","status":200,"time":"2022-01-04T14:51:53Z","user-agent":"Prometheus/2.19.2"} {"fields.time":"2022-01-04T14:51:54Z","ip":"10.46.0.0","latency":850,"level":"info","method":"GET","msg":"","path":"/healthz","status":404,"time":"2022-01-04T14:51:54Z","user-agent":"kube-probe/1.19"}

[rambabubolla]

promothisys config:

remote_write:

• url: http://prometheus-kafka-adapter:8080/receive remote_timeout: 30s queue_config: capacity: 500 max_shards: 1000 min_shards: 1 max_samples_per_send: 100 batch_send_deadline: 5s min_backoff: 30ms max_backoff: 100ms

ncn-m001:~ # kubectl get svc

prometheus-kafka-adapter ClusterIP 10.22.149.193 80/TCP 5h

[palmerabollo]

@rambabubolla The new health endpoint was included in https://github.com/Telefonica/prometheus-kafka-adapter/pull/71/files and merged on 8 Jun 2021. The latest prometheus-kafka-adapter version (1.7.0) was released on 4 Jun 2021, so it doesn't include that feature yet.

As a workaround, please use /metrics as your health endpoint in your kubernetes manifests until we release a new 1.8.0 docker image (I'll try to do it this week).

Thanks.

[rambabubolla]

@palmerabollo thanks a lot 👍 it is working

[palmerabollo]

@rambabubolla There is a new release 1.8.0 that includes the new /healthz endpoint, just in case you find it useful.

[rambabubolla]

@rambabubolla There is a new release 1.8.0 that includes the new /healthz endpoint, just in case you find it useful.

@palmerabollo thanks for sharing .For me unable to get kafaka data raised the issue