search

enix / x509-certificate-exporter

A Prometheus exporter to monitor x509 certificates expiration in Kubernetes clusters or standalone
MIT License
631 stars 64 forks source link

Ability to set QPS and Burst to avoid client-side request throttling #278

Closed age9990 closed 4 months ago

age9990 commented 5 months ago

In our k8s cluster, there are hundreds of namespaces. As a result, it takes a lot of time to query secrets of certificates. What's more, when multiple scraping occurred, warning message of client-side request throttling showed in the logs. The message is like this, "Waited for Xs due to client-side throttling, not priority and fairness". From https://github.com/kubernetes/client-go/blob/aa7909e7d7c0661792ba21b9e882f3cd6ad0ce53/rest/config.go#L116 we can see the default value of QPS and burst are 5 and 10 respectively for go k8s client, which may be not enough for a large-scale cluster. I tried to increase QPS and burst by setting the following code in function connectToKubernetesCluster() config.RateLimiter = flowcontrol.NewTokenBucketRateLimiter(50, 100) The time needed the first time querying all secrets gradually decreased from 150s to 10s, and the subsequent metrics scraping time is decreased from 30s to 1s. It would be great if there are options to set QPS and burst as parameters.

monkeynator commented 4 months ago

:tada: This issue has been resolved in version 3.14.0-beta.1 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket:

monkeynator commented 4 months ago

:tada: This issue has been resolved in version 3.14.0 :tada:

The release is available on GitHub release

Your semantic-release bot :package::rocket: