mvisonneau / gitlab-ci-pipelines-exporter

Prometheus / OpenMetrics exporter for GitLab CI pipelines insights
Apache License 2.0
1.27k stars 240 forks source link

Exporter stops responding but process keeps consuming resources #869

Open pedroosorio opened 3 months ago

pedroosorio commented 3 months ago

Hi,

I've deployed the exporter in a Ubuntu 22.04 machine running Docker version 20.10.22, build 3a2c30b and the latest exporter version. We are using wildcards to grab everything from the project:

Our metrics are growing exponentially - that is probably due to bad configuration for sure - but the exporter is behaving in an unexpected manner:

image

As we can see, it stops comunicating with Prometheus, but still consumes CPU and RAM. I would expect that under a crashing condition - also logs don't reveal anything - the process would stop, restarting the container.

Also, given your experience, for this amount of refs, does it warrant the HA deployment instead ?