search

mvisonneau / gitlab-ci-pipelines-exporter

Prometheus / OpenMetrics exporter for GitLab CI pipelines insights
Apache License 2.0
1.26k stars 239 forks source link

Queue buffer size exhausted #648

Open ceprateek opened 1 year ago

ceprateek commented 1 year ago

I keep seeing this in the logs for the exporter: {"level":"warning","msg":"queue buffer size exhausted, skipping scheduling of task..","task_type":"PullRefMetrics","task_unique_id":"1760622161","time":"2023-05-23T20:29:32Z"}

Can you please let me know what is the impact. I see in the code that the size is set to 1000 and there is no option to change it without forking the repo

Routhinator commented 10 months ago

I guess the workaround is multiple exporters and targetting smaller subgroups of projects, but seeing this as well.

Routhinator commented 10 months ago

Well I found the implication @ceprateek - the agent is effectively dead and produces zero metrics. The only way out of this seems to be to manually restart the process....

This needs some fixing it would seem...

Routhinator commented 10 months ago

@mvisonneau I am guessing the only workaround for now that is stable is to deploy multiple exporters scoped to smaller project groups and it removes the ability to use the wildcard "all projects" setting...

Do you have any explanation on why the 1000 setting was hardcoded? Could this be changed without harming the application?

R-Studio commented 8 months ago

We have the same issue with a maximum_jobs_queue_size of 5000. Any news or tips how to scale the exporter for big GitLab instances?

@ceprateek I think you can change the setting: https://github.com/mvisonneau/gitlab-ci-pipelines-exporter/blob/0687efd1075d33283c3762a135aa24de94ffe21c/docs/configuration_syntax.md?plain=1#L106C21-L106C21

ehonda commented 5 months ago

There's also an explanation what can be configured to mitigate this here