search

emeraldpay / dshackle

Fault Tolerant Load Balancer for Ethereum and Bitcoin APIs
Apache License 2.0
308 stars 55 forks source link

Metrics server crashes after some time #282

Closed eskp closed 6 months ago

eskp commented 6 months ago

We have Prometheus metrics enabled and accessible locally on the container itself and when queried on the remote Prometheus. But after some time these metrics stop getting pushed. When going on the container and running curl localhost:8081/metrics we get a timeout. The only metrics related log is MonitoringSetup | Run Prometheus metrics on 0.0.0.0:8081/metrics.

splix commented 6 months ago

Does it respond to any other requests? Like proxy endpoint, if it's enabled, or the health checks?

eskp commented 6 months ago

Does it respond to any other requests? Like proxy endpoint, if it's enabled, or the health checks?

Yes the proxy functionality continues to work and /health endpoint returns OK

splix commented 6 months ago

Do you have any errors in the logs? Or anything else that may suggest the source of the problem?

eskp commented 6 months ago

@splix no errors in logs, is there debug logs that I can turn on?

splix commented 6 months ago

You can run it with debug logging by setting env DSHACKLE_LOG_LEVEL=DEBUG but I don't think it could help with this particular problem. Because errors are printed to console by default.

Anyway, do I understand correctly that it's a reproducible problem with your setup? I mean, do you have it each time you run Dshackle? Or happened just once?

eskp commented 6 months ago

Closing this for now as this hasn't happened in the past week.