Prometheus Exporter client is dropping message cause queue is full

[leshik]

Hi,

What might be the reason for this message? The collector is running on port 9394 and seems to work ok, but I see these messages in the app log under load.

[SamSaffron]

Do you have a bit more details about what you are collecting / how many metrics?

[leshik]

Hi,

Sure. The following files are used when building the prometheus_exporter container:

Gemfile:

source 'https://rubygems.org'

gem 'graphql', '~> 1.8.0'
gem 'prometheus_exporter'

appinfo_collector.rb:

# frozen_string_literal: true

class AppInfoCollector < PrometheusExporter::Server::TypeCollector
  MAX_APPINFO_METRIC_AGE = 60

  def initialize
    @appinfo_metrics = []
  end

  def type
    'appinfo'
  end

  def metrics
    return [] if @appinfo_metrics.length == 0

    gauge = PrometheusExporter::Metric::Gauge.new('app', 'Application information')

    @appinfo_metrics.map do |metric|
      labels = metric.select { |label| ['myapp_version', 'ruby_version'].include?(label) }
      gauge.observe(1, labels)
    end

    gauge
  end

  def collect(obj)
    now = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)

    obj['created_at'] = now

    @appinfo_metrics.delete_if do |current|
      obj['myapp_version'] == current['myapp_version'] || (current['created_at'] + MAX_APPINFO_METRIC_AGE < now)
    end

    @appinfo_metrics << obj
  end
end

graphql_collector.rb:

# frozen_string_literal: true

require 'graphql/tracing'

class GraphQLCollector < GraphQL::Tracing::PrometheusTracing::GraphQLCollector
end

It then launched as bundle exec prometheus_exporter -a appinfo_collector.rb -a graphql_collector.rb.

In the application I have:

config/initializers/metrics.rb:

require 'prometheus_exporter'
require 'prometheus_exporter/client'

client = PrometheusExporter::Client.new(
  host: ENV['PROMETHEUS_EXPORT_COLLECTOR_HOST'],
  port: PrometheusExporter::DEFAULT_PORT,
)

PrometheusExporter::Client.default = client

require 'prometheus_exporter/middleware'
Rails.application.middleware.unshift PrometheusExporter::Middleware

require 'prometheus_exporter/instrumentation'
PrometheusExporter::Instrumentation::Process.start(type: 'master')

config/initializers/okcomputer.rb:

class AppVersionCheck < OkComputer::Check
  def check
    app_version_file = Rails.root + 'public/version'
    if File.exist?(app_version_file)
      app_version = File.read(app_version_file).strip
      PrometheusExporter::Client.default.send_json(
        type: 'appinfo',
        myapp_version: app_version,
        ruby_version: RUBY_VERSION,
      )
      mark_message "MyApp version: #{app_version}"
    else
      mark_failure
      mark_message "File doesn't exist: #{app_version_file}"
    end
  end
end

OkComputer::Registry.register 'myapp_version', AppVersionCheck.new

config/initializers/sidekiq.rb:

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    require 'prometheus_exporter/instrumentation'
    chain.add PrometheusExporter::Instrumentation::Sidekiq
  end

  config.on :startup do
    require 'prometheus_exporter/instrumentation'
    PrometheusExporter::Instrumentation::Process.start type: 'sidekiq'
  end
end

config/puma.rb:

...
on_worker_boot do
  ActiveSupport.on_load(:active_record) do
    ActiveRecord::Base.establish_connection
  end

  require 'prometheus_exporter/instrumentation'
  PrometheusExporter::Instrumentation::Process.start(type: 'web')
end
...

app/graph/schema.rb:

class Schema < GraphQL::Schema
  query QueryType
  mutation MutationType
  lazy_resolve(Promise, :sync)
  use GraphQL::Batch
  use(GraphQL::Tracing::PrometheusTracing)
end

Schema.graphql_definition

And that's it.