search

joe-elliott / cert-exporter

A Prometheus exporter that publishes cert expirations on disk and in Kubernetes secrets
Apache License 2.0
322 stars 87 forks source link

During cert exporter bootup, cert-exporter consuming 143+ MB of memory #137

Open dsnr-coder opened 1 year ago

dsnr-coder commented 1 year ago

Hi,

In the past few helm installations of cert exporter, we could see that cert exporter memory spike is happening and consuming memory around 143+ MB.

We have not modified resource limits that were present in cert-exporter helm chart. With the below resource limits, helm installations where going through since for sometime, all of a sudden, since past 5 installations, intermittently we started observing that cert-exporter is going into CrashLoopBackOff state, due to OOMKilled error.

When I check Grafana for cert exporter memory usage, I could see that for 3 seconds cert exporter has consumed 143+ MB, after 3 seconds, it has settled down, and not went beyond 90MB of memory.

  resources:
     limits:
       cpu: 100m
       memory: 128Mi
     requests:
       cpu: 100m
       memory: 128Mi

I had looked into the logs of cert-exporter, I could see that periodicSecretChecker.go took around 8 seconds , when periodicSecretChecker is getting executed, cert-exporter is consuming more memory. image

Please find the screenshot from Grafana below image

I would appreciate if you could please throw some light on guidance on how to troubleshoot and fix the issue.

joe-elliott commented 1 year ago

If you can profile memory and provide a flamegraph that would help identify memory usage. Other useful information would be the kind and number of secrets you are looking for.