During our rails app shutdown we got lot of errors like this:
Prometheus Exporter, failed to send message Connection refused - connect(2) for \"localhost\" port 9394
This is happening because the metrics sidecar is faster than our up in shutdown, so it finishes before us.
I added a preStop hook in our app deployment that performs (more or less) the following ruby code:
logger.info "Stopping PrometheusExporter and relevant instrumentation"
# Stop the periodic stats, stop is a noop if instrumentation didn't start thus is safe to
# call it for every instrumentation we're using
PrometheusExporter::Instrumentation::Puma.stop
PrometheusExporter::Instrumentation::ActiveRecord.stop
PrometheusExporter::Instrumentation::Process.stop
PrometheusExporter::Instrumentation::SidekiqProcess.stop
PrometheusExporter::Instrumentation::SidekiqQueue.stop
PrometheusExporter::Instrumentation::SidekiqStats.stop
# Finally also stop the client, just to be safe
PrometheusExporter::Client.default.stop(wait_timeout_seconds: 10)
logger.info "Stopped all Prometheus instrumentations"
But I still get 2 or 3 messages after Stopped all Prometheus instrumentations log message and that sounds very weird to me. Am I doing something wrong? Is this the proper way to stop the exporter?
Here's a snippet from our logfiles.
timestamp;level;message;name
2024-10-18T14:18:25.608302Z;error;"Prometheus Exporter, failed to send message Connection refused - connect(2) for ""localhost"" port 9394";PrometheusExporter::Client
2024-10-18T14:18:25.107240Z;error;"Prometheus Exporter, failed to send message Connection refused - connect(2) for ""localhost"" port 9394";PrometheusExporter::Client
2024-10-18T14:18:24.606151Z;error;"Prometheus Exporter, failed to send message Connection refused - connect(2) for ""localhost"" port 9394";PrometheusExporter::Client
2024-10-18T14:18:24.603777Z;info;Completed #shutdown;Api::Internal::ShutdownController
2024-10-18T14:18:24.530111Z;info;Stopped all Prometheus instrumentations;MetricsExporter
2024-10-18T14:18:24.528236Z;info;Requested shutdown by internal API;Api::Internal::ShutdownController
During our rails app shutdown we got lot of errors like this:
This is happening because the metrics sidecar is faster than our up in shutdown, so it finishes before us.
I added a
preStophook in our app deployment that performs (more or less) the following ruby code:But I still get 2 or 3 messages after
Stopped all Prometheus instrumentationslog message and that sounds very weird to me. Am I doing something wrong? Is this the proper way to stop the exporter?Here's a snippet from our logfiles.