search

prometheus / exporter-toolkit

Utility package to build exporters
Apache License 2.0
268 stars 81 forks source link

Dedicated webserver for health check endpoint #111

Open tejaswiniVadlamudi opened 2 years ago

tejaswiniVadlamudi commented 2 years ago

Prometheus with direct TLS support (by using --web.config.file flag) expects strict client certificates for all HTTP endpoint consumers. When deployed on k8s, k8s (kubelet) doesn't provide a client certificate when it probes readiness and liveness endpoints. One needs to disable health check probes in the deployment manifests in order to deploy Prometheus with inbuilt TLS support on k8s.

See k8s documentation here (https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#http-probes). It says "'if the scheme field is set to HTTPS, the kubelet sends an HTTPS request skipping the certificate verification"

Can we move /-/ready and /-/healthy endpoints to a separate web server?

roidelapluie commented 2 years ago

It would be acceptable to me to be able to somehow allow /-/ready and /-/healthy to not check for certificate.

tejaswiniVadlamudi commented 2 years ago

@roidelapluie : Do you accept code contributions for this task? If yes, could you hint me at the next step?

roidelapluie commented 2 years ago

On 08 Jun 15:42, tejaswiniVadlamudi wrote:

@roidelapluie : Do you accept code contributions for this task? If yes, could you hint me at the next step?

-- Reply to this email directly or view it on GitHub: https://github.com/prometheus/exporter-toolkit/issues/111 You are receiving this because you were mentioned.

Message ID: @.***>

The next step would be a pull request on the exporter-toolkit repository with an option: "allow_anonymous_probes: true" to enable readiness and liveness probes without authentication.

-- Julien Pivotto O11y - https://o11y.eu/

roidelapluie commented 2 years ago

We are working on this https://github.com/prometheus/exporter-toolkit/pull/106

twiden commented 2 years ago

I think it would be really good to be able to exclude /-/healthy and /-/ready from basic auth and possibly TLS verification as well. My usecase is that I run Pushgateway on ECS and basic auth prevents the Application Load Balancer to perform meaningful health checks. I also only push metrics to the gateway on the local subnet so in my case a good solution would be to bypass basic auth for health, readiness and metrics push and maybe add ip whitelisting based on cidr for those endpoints instead.