harmjanblok / puma-metrics

Puma plugin to export puma stats as prometheus metrics
MIT License
94 stars 27 forks source link

Deal with server not ready gracefully #68

Open dgsuarez opened 1 week ago

dgsuarez commented 1 week ago

Fixes https://github.com/harmjanblok/puma-metrics/issues/49

We've also observed this situation sometimes.

This implements a simple solution, the response is still an error so prometheus should handle it as such, but an exception doesn't bubble up in the app.

I haven't been able to find a way to query any public interface offered by puma to identify the situation, so I'm capturing the exception.

Testing is not straightforward since it depends on timing. One way to do it is to insert a sleep 10 here:

https://github.com/puma/puma/blob/8e7fab5f3992ef056dd9030a89da0372f933e5d4/lib/puma/single.rb#L51

With that, if we make a metrics request right after the app starts we see this code in action:

 $ curl -i localhost:9393
HTTP/1.1 503 Service Unavailable
Content-Type: text/plain
Content-Length: 45

Puma is booting up. Stats are not yet ready.

After 10 seconds requests return the regular prometheus payload:

$ curl -i localhost:9393
HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 926

# TYPE puma_backlog gauge
...
dgsuarez commented 3 days ago

Hey @harmjanblok can you take a look at this one? Thanks!