joe-elliott / cert-exporter

A Prometheus exporter that publishes cert expirations on disk and in Kubernetes secrets
Apache License 2.0
321 stars 87 forks source link

chore: add check count metric for secrets #58

Closed one1zero1one closed 3 years ago

one1zero1one commented 4 years ago

Trying to address https://github.com/joe-elliott/cert-exporter/issues/56, how to ensure alerts go away when you delete an expired certificate stored in a secret.

Assuming the pooling frequency is 15m counting each time a certificate is checked, allows us to to alert on things like (cert_exporter_secret_expires_in_seconds <= 604800) * (rate(cert_exporter_secret_check_total[30m]) > bool 0) in other words - if the certificate expires soon and it has been checked.

Not sure if that's the best way, but it works on our local tests.

joe-elliott commented 4 years ago

I suppose this is one way to do it. I'd hope for a way to just stop exposing the metric.

@beorn7 is there a way in client_golang to stop exposing a metric? In this case the expiration date of an x509 cert that is stored in a k8s secret is being exported. After the secret disappears the metric remains but is no longer updated. Is there a way to drop this metric?

beorn7 commented 4 years ago

You can unregister a collector with Unregister: https://pkg.go.dev/github.com/prometheus/client_golang@v1.8.0/prometheus#Registerer

Or you can delete an individual metric from a MetricVec: https://pkg.go.dev/github.com/prometheus/client_golang@v1.8.0/prometheus#MetricVec.Delete

joe-elliott commented 4 years ago

Thanks @beorn7

@one1zero1one These look a little awkward, but I'd rather attempt to use them to just remove metrics for certs that no longer exist.

Maybe we could just clear all cert metrics at the start of the check loop?

joe-elliott commented 3 years ago

This was fixed in #79