Confused about custom metrics

[vitobotta]

Hi! Thanks a lot for this immensely useful tool :)

I am trying to add metrics from https://github.com/DmitryTsepelev/io_monitor - it can publish data to ActiveSupport notifications so I am trying to expose memory info per endpoint as metrics to Prometheus. I am trying to find out which speciendpoints use the most memory because we seem to have a problem with growing memory usage that seems to be triggered randomly.

At the moment I have this in an initializer:

IoMonitor.configure do |config|
  config.publish = [:logs, :notifications] # defaults to :logs
  config.warn_threshold = 0.8 # defaults to 0
  config.adapters = [:active_record, :net_http, :redis] # defaults to [:active_record]
end

if Rails.env.production?
  require 'prometheus_exporter/client'

  client = PrometheusExporter::Client.new(host: '127.0.0.1', port: 9394)

  ActiveSupport::Notifications.subscribe("process_action.action_controller") do |name, start, finish, id, payload|

    io_monitor = payload[:io_monitor] || {}# { active_record: 866, response: 25 }

    metrics = payload.slice(:controller, :action, :method, :status, :view_runtime, :db_runtime, :path).merge(io_monitor)

    client.send_json(metrics)
  end
end

But I don't see these metrics in Prometheus. Do I need a custom collector? sorry, I read the README but I am still confused. All I would like to see is the most offending endpoints for memory usage. Thanks!

[SamSaffron]

Yeah, if you are publishing custom metrics you need something to collect them.

[axsuul]

I personally don't use custom collectors, I just send the metrics directly to the prometheus_exporter server from Sidekiq