We've observed some issues with the Pushgateway when handling concurrent requests:
Issue: When we start the systemd services for the network concurrently, it appears that some metrics are not pushed as expected. This may lead to incomplete or missing data being recorded in Prometheus.
Proposed Investigation:
We should explore potential solutions to manage the concurrent start of these services more effectively.
To better understand the situation, we need to extract the average time required to complete the task on each network from Grafana.
With the data gathered, we can interleave the service start times, staggering them to avoid overloading the Pushgateway with concurrent requests.
This approach should help in ensuring that metrics are reliably pushed and recorded for all network services.
albertodvp
commented
2 weeks ago
Issue Description:
We've observed some issues with the Pushgateway when handling concurrent requests:
Issue: When we start the systemd services for the network concurrently, it appears that some metrics are not pushed as expected. This may lead to incomplete or missing data being recorded in Prometheus.
Proposed Investigation:
This approach should help in ensuring that metrics are reliably pushed and recorded for all network services.