Open vitobotta opened 2 years ago
I have the opposite problem. I'm getting the main metrics, but not the worker metrics.
My puma.rb
looks like yours, but I also have some configuration in an initializer
to run the server and start the process for the "main" metrics:
initializers/prometheus.rb
if Rails.env == "production"
require 'prometheus_exporter/server'
require 'prometheus_exporter/client'
require 'prometheus_exporter/instrumentation'
require 'prometheus_exporter/middleware'
# This reports stats per request like HTTP status and timings
Rails.application.middleware.unshift PrometheusExporter::Middleware
server = PrometheusExporter::Server::WebServer.new bind: 'localhost', port: ENV.fetch("METRICS_PORT") { 9090 }
server.start
# # wire up a default local client
PrometheusExporter::Client.default = PrometheusExporter::LocalClient.new(collector: server.collector)
# this reports basic process stats like RSS and GC info
PrometheusExporter::Instrumentation::Process.start(type: "main")
end
Where do you see your web worker metrics? Do you also run the metrics server like I do?
Hi @NickLarsenNZ and sorry for the delay, I never received a notification of your reply and I only noticed it now because I am still investigating this issue. I see only some metrics in Prometheus, specifically workers, old workers and booted workers. The one I am most interested in is the backlog and I never got that metric to show up. I am not sure if I am still missing some configuration....
In the initializer I have this:
if Rails.env.production?
require 'prometheus_exporter/middleware'
require 'prometheus_exporter/server'
require 'prometheus_exporter/client'
require 'prometheus_exporter/instrumentation'
Rails.application.middleware.unshift PrometheusExporter::Middleware
end
I'm gonna add the start line in there as you do, perhaps that will give me the backlog?
It seems I can get the request backlog if I use before_fork instead of on_worker_boot, which one are you using?
@vitobotta You need to place your instrumentation code into after_worker_boot
section if running Puma in clustered mode.
For anyone trying to get it working generally (for both standalone and clustered mode), I've done the following, but have not tested it properly (this was from months ago), but maybe it helps.
config/puma.rb
if ENV.fetch("PROMETHEUS_EXPORTER_ENABLED", "false").downcase == "true"
# Puma is a bit weird with configuration between standalone and clustered mode.
# When workers is 0, then the main process does the threading.
# But there are no hooks for configuring that. Maybe we should force at least
# one worker to make use of the after_worker_boot hook
# we have to wrap this if workers are 0 or undefined, else we miss the following metrics:
# HELP puma_running_threads Number of puma threads currently running.
# TYPE puma_running_threads gauge
# puma_running_threads{phase="0",app="rails-metrics"} 10
# puma_request_backlog{phase="0",app="rails-metrics"} 0
# puma_thread_pool_capacity{phase="0",app="rails-metrics"} 10
# puma_max_threads{phase="0",app="rails-metrics"} 10
if @options.fetch(:workers) { 0 } == 0
PrometheusExporter::Instrumentation::Process.start(type: "main")
# E, [2022-09-06T12:13:59.369144 #76882] ERROR -- : PrometheusExporter::Instrumentation::Puma Prometheus Exporter Failed To Collect Stats undefined method `stats' for nil:NilClass
# Issue: https://github.com/puma/puma/issues/1230
# PR: https://github.com/puma/puma/pull/2709
# Still seems to work, so leaving in here.
PrometheusExporter::Instrumentation::Puma.start(frequency: 1)
PrometheusExporter::Instrumentation::ActiveRecord.start(
custom_labels: { type: "puma_standalone_mode" }, #optional params
config_labels: [:database, :host] #optional params
)
end
after_worker_boot do
PrometheusExporter::Instrumentation::Process.start(type: "web")
PrometheusExporter::Instrumentation::ActiveRecord.start(
custom_labels: { type: "puma_clustered_mode" }, #optional params
config_labels: [:database, :host] #optional params
)
# if this is started outside after_worker_boot, then some metrics disappear
if !PrometheusExporter::Instrumentation::Puma.started?
PrometheusExporter::Instrumentation::Puma.start(frequency: 1)
end
end
end
config/prometheus.rb
Note: I'm unsure if it's ok to do the
PrometheusExporter::Instrumentation::Process.start
again when it's done in puma.rb also. But think it is needed to get Delayed Job metrics (in which case you need to avoid running the local server, and send to a separateprometheus_exporter
process.
if ENV.fetch("PROMETHEUS_EXPORTER_ENABLED", "false").downcase == "true"
require 'prometheus_exporter/server'
require 'prometheus_exporter/client'
require 'prometheus_exporter/instrumentation'
require 'prometheus_exporter/middleware'
# This reports stats per request like HTTP status and timings
Rails.application.middleware.unshift PrometheusExporter::Middleware
# You can run the local server
if ENV.fetch("PROMETHEUS_EXPORTER_LOCAL_SERVER_ENABLED", "false").downcase == "true"
server = PrometheusExporter::Server::WebServer.new(
bind: ENV.fetch("PROMETHEUS_EXPORTER_HOST") { PrometheusExporter::DEFAULT_BIND_ADDRESS },
port: ENV.fetch("PROMETHEUS_EXPORTER_PORT") { PrometheusExporter::DEFAULT_PORT }
)
server.start
# wire up a default local client
PrometheusExporter::Client.default = PrometheusExporter::LocalClient.new(collector: server.collector)
end
PrometheusExporter::Metric::Base.default_labels = { "app" => "cymonz-web" }
# this reports basic process stats like RSS and GC info
PrometheusExporter::Instrumentation::Process.start(type: "main")
end
After tested in my env, I found that if using after_worker_boot
, can only see the master's ruby_rss
. In order to see the worker's ruby_rss
, you can try to use the on_worker_boot
.
I amended a bit from @NickLarsenNZ version, config/puma.rb
require 'prometheus_exporter/instrumentation'
if @options.fetch(:workers) { 0 } == 0
PrometheusExporter::Instrumentation::Process.start(type: "master")
PrometheusExporter::Instrumentation::Puma.start(frequency: 1)
PrometheusExporter::Instrumentation::ActiveRecord.start(
custom_labels: { type: "puma_single_mode" }, #optional params
config_labels: [:database, :host] #optional params
)
else
PrometheusExporter::Instrumentation::Process.start(type: "main")
PrometheusExporter::Instrumentation::Puma.start(frequency: 1)
end
on_worker_boot do
PrometheusExporter::Instrumentation::Process.start(type: "web")
PrometheusExporter::Instrumentation::ActiveRecord.start(
custom_labels: { type: "puma_clustered_mode" }, #optional params
config_labels: [:database, :host] #optional params
)
# if this is started outside after_worker_boot, then some metrics disappear
if !PrometheusExporter::Instrumentation::Puma.started?
PrometheusExporter::Instrumentation::Puma.start(frequency: 1)
end
end
The reason why the master
for single mode, main
for cluster mode, it is because the grafana dashboard I used, the web workers
querying the master|web|puma_master|puma_worker
, I didn't want the master to be count as a worker when cluster mode but I did want to see the rest of the metrics of the master in the dashboard. So when in single mode, the master can be count as a web worker
I could get it to work. For those interested:
highlights:
on_booted
blockafter_worker_boot
unless PrometheusExporter::Instrumentation::Puma.started?
else you'll only get metrics for a single workerIt all seems to work. probably the doc needs improving though.
Hi!
I have configured the Puma instrumentation as per the README but I only see the worker related metrics. I would need to see puma_request_backlog for example but it doesn't show up in Prometheus. What am I missing? Thanks!
ps. This is in my puma.rb