search

discourse / prometheus_exporter

A framework for collecting and aggregating prometheus metrics
MIT License
538 stars 155 forks source link

After using PrometheusExporter::Middleware, my rails app APIs throwing 502s and pods are crashing #296

Closed the-blackbeard closed 11 months ago

the-blackbeard commented 12 months ago

I was using prometheus exporter to collect the metrics. I recently wanted to use the metrics related to the APIs so I added PrometheusExporter::Middleware in my config. And ever since some of my applications APIs started giving 502s and the pods started crashing or going OOM. I just added this line: 

  # This reports stats per request like HTTP status and timings
  Rails.application.middleware.unshift PrometheusExporter::Middleware

If I remove this line of code from my config file, all the APIs work perfectly but as soon as I add this line some of my APIs start to give 502s. The APIs are mostly those APIs which are making changes in the DB, so all the PUT/POST APIs are giving 502s and all the GET APIs are working perfectly fine. Can someone help me in resolving this issue please?

the-blackbeard commented 11 months ago

I fixed this by using this

Rails.application.middleware.unshift PrometheusExporter::Middleware, instrument: :prepend