Troubleshooting a Rails/Unicorn/Sidekiq setup

[jlecour]

Hi,

I'm trying to use Metriks in a Rails 4.1 app.

I've added an initializer to start a Librato reporter, and I've added a few ActiveSupport::Notification subscriptions to collect data and emit some metrics.

I also have Sidekiq to manage a queue of jobs with a few direct Metriks emissions (not through AS::N).

On my development system, it works really fine ; I see my various metrics in Librato. But in staging, I don't have all the data in Librato.

If I keep the reporter set up in the initializer, the Sidekiq process sends its metrics fine, but the Unicorn workers don't. I've figured out that it's because of the forking model of Unicorn.

If I put the reporter's initialization in the after_fork block of Unicorn's config file, I have some data from my Rails processes, but not from Sidekiq.

I didn't find a good way of keeping my Metriks reporter configuration between Rails regular processes (console, rake, Sidekiq workers, …) and the application server's workers (Unicorn in my case, Passenger, …).

With ActiveRecord, there is a class method to re-establish the connection in Unicorn's special case, but there is no configuration. I'd like to end up with a similar thing where the reporter is set up in a regular way and simply restarted when needed (after a fork).

Thanks for any help of insight.

[eric]

You're right about the issue with unicorn (or anything that forks like resque).

You'll need to call .restart (or .stop and .start) on your reporters after forking. 

The reason for this is threads do not survive across forks.

On Tue, Apr 15, 2014 at 10:57 AM, Jérémy Lecour notifications@github.com wrote:

Hi, I'm trying to use Metriks in a Rails 4.1 app. I've added an initializer to start a Librato reporter, and I've added a few ActiveSupport::Notification subscriptions to collect data and emit some metrics. I also have Sidekiq to manage a queue of jobs with a few direct Metriks emissions (not through AS::N).
On my development system, it works really fine ; I see my various metrics in Librato. But in staging, I don't have all the data in Librato. If I keep the reporter set up in the initializer, the Sidekiq process sends its metrics fine, but the Unicorn workers don't. I've figured out that it's because of the forking model of Unicorn. If I put the reporter's initialization in the after_fork block of Unicorn's config file, I have some data from my Rails processes, but not from Sidekiq. I didn't find a good way of keeping my Metriks reporter configuration between Rails regular processes (console, rake, Sidekiq workers, …) and the application server's workers (Unicorn in my case, Passenger, …). With ActiveRecord, there is a class method to re-establish the connection in Unicorn's special case, but there is no configuration. I'd like to end up with a similar thing where the reporter is set up in a regular way and simply restarted when needed (after a fork).

Thanks for any help of insight.

Reply to this email directly or view it on GitHub: https://github.com/eric/metriks/issues/45

[jlecour]

I've managed to make this work, but I'd like feedback.

In an config/initializers/metriks.rb :

if Settings.librato.email && Settings.librato.api_key
  require 'metriks/reporter/librato_metrics'
  $metrics_reporter = Metriks::Reporter::LibratoMetrics.new(
    Settings.librato.email,
    Settings.librato.api_key
  )
else
  require 'metriks/reporter/logger'
  $metrics_reporter = Metriks::Reporter::Logger.new(
    logger: Logger.new(Rails.root.join("log","metrics.log")),
  )
end

if $metrics_reporter
  $metrics_reporter.start
  Rails.logger.info("#{$metrics_reporter.class.name} start")
end

then in config/unicorn/production.rb :

(…)
after_fork do |server, worker|
  if $metrics_reporter
    $metrics_reporter.restart 
    Rails.logger.info("#{$metrics_reporter.class.name} restart")
  end
end
(…)

I use a fallback to a Logger reporter, just in case, or for stages that are not supposed to push their data to Librato. I (temporarily) log the start and restart action in the Rails logger, but it's completely optional.

What I'm not confident about is the use of a global variable for the reporter. It looks like the reporter is thread safe, but I might be wrong.

[eric]

I think that code looks perfect.

I have something very similar and use a global as well.