Sidekiq errors and exits during startup when using datadog and prometheus exporter

dorongutman commented 2 years ago

Ruby version: 2.7.1 Rails version: 6.0.4.4

I am cross posting this in the other gems repos as I don't know who/what is the culprit. When trying to run sidekiq with both prometheus exporter and datadog's sdk (ddtrace), sidekiq exits upon start (after a few seconds of waiting) with the following error: stack level too deep

Full traceback log:

Traceback (most recent call last):
    8741: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/ruby_executable_hooks:22:in `<main>'
    8740: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/ruby_executable_hooks:22:in `eval'
    8739: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/sidekiq:23:in `<main>'
    8738: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/sidekiq:23:in `load'
    8737: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/bin/sidekiq:31:in `<top (required)>'
    8736: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq/cli.rb:70:in `run'
    8735: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq.rb:117:in `redis_info'
    8734: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq.rb:97:in `redis'
     ... 8729 levels...
       4: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/extensions.rb:106:in `[]'
       3: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/configurable.rb:51:in `resolve'
       2: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/configuration/resolver.rb:14:in `resolve'
       1: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/configuration/resolver.rb:22:in `parse_matcher'
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/vendor/resolver.rb:39:in `resolve': stack level too deep (SystemStackError)

This issue doesn't happen without the datadog gem or the prometheus gem.

I have created a reproduction repo with as little changes as possible compared to a fresh rails new installation - https://github.com/dorongutman/rails-sidekiq-datadog-prometheus-issue

The readme explains everything but in short:

the only changes made are the addition of the three gems, and 2 initializers - promethus.rb and sidekiq.rb - both only have prometheus-exporter relevant configuration, as described in the prometheus exporter docs.
removing the middleware configuration of the prometheus exporter makes sidekiq load fine without the error

For convenience, here are the two initializers: sidekiq.rb

require 'sidekiq/api'
require 'prometheus_exporter/instrumentation'

$LOAD_PATH << Rails.root

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    chain.add PrometheusExporter::Instrumentation::Sidekiq
  end
  config.death_handlers << PrometheusExporter::Instrumentation::Sidekiq.death_handler
  config.on :startup do
    PrometheusExporter::Instrumentation::SidekiqQueue.start
    PrometheusExporter::Instrumentation::Process.start type: 'sidekiq'
  end
  at_exit do
    PrometheusExporter::Client.default.stop(wait_timeout_seconds: 10)
  end
end

prometheus.rb

  require 'prometheus_exporter/middleware'
  Rails.application.middleware.unshift PrometheusExporter::Middleware

Here's the full log from running sidekiq to its death (this specific run was with RAILS_ENV=production and DD_TRACE_ENABLED=false to have fewer lines and potential issues but it happens in any combination):

 bundle exec sidekiq
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:115: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:119: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:123: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:146: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:150: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:154: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:288: warning: The called method `initialize' is defined here
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:84: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/configuration/agent_settings_resolver.rb:32: warning: The called method `initialize' is defined here
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: Booted Rails 6.0.4.4 application in production environment
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: Running in ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [-darwin21]
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org
2022-01-28T07:47:30.605Z pid=48210 tid=yv6 INFO: Booting Sidekiq 6.4.0 with redis options {}
Traceback (most recent call last):
    8741: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/ruby_executable_hooks:22:in `<main>'
    8740: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/ruby_executable_hooks:22:in `eval'
    8739: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/sidekiq:23:in `<main>'
    8738: from /Users/doron/.rvm/gems/ruby-2.7.1/bin/sidekiq:23:in `load'
    8737: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/bin/sidekiq:31:in `<top (required)>'
    8736: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq/cli.rb:70:in `run'
    8735: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq.rb:117:in `redis_info'
    8734: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/sidekiq-6.4.0/lib/sidekiq.rb:97:in `redis'
     ... 8729 levels...
       4: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/extensions.rb:106:in `[]'
       3: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/configurable.rb:51:in `resolve'
       2: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/configuration/resolver.rb:14:in `resolve'
       1: from /Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/configuration/resolver.rb:22:in `parse_matcher'
/Users/doron/.rvm/gems/ruby-2.7.1/gems/ddtrace-0.54.2/lib/ddtrace/contrib/redis/vendor/resolver.rb:39:in `resolve': stack level too deep (SystemStackError)

SamSaffron commented 2 years ago

This is the old method patching problem. You can pick aliasing method chains or prepending modules. Start mixing them together and trouble starts happening.

Open to a PR that allows you to opt for a style of method patching as long as we keep the current default.

inner-whisper commented 2 years ago

@SamSaffron I think this issue can be closed by same reason as https://github.com/discourse/prometheus_exporter/issues/159#issuecomment-1146203248

Style of method patching can be chosen now - https://github.com/discourse/prometheus_exporter#choosing-the-style-of-method-patching

SamSaffron commented 2 years ago

Thanks @inner-whisper !

discourse / prometheus_exporter

Sidekiq errors and exits during startup when using datadog and prometheus exporter #212