search

BaseSecrete / rorvswild

Performances & exceptions monitoring for Ruby on Rails applications
https://www.rorvswild.com
MIT License
331 stars 9 forks source link

Conflict with Datadog #30

Closed GlauberrBatista closed 3 months ago

GlauberrBatista commented 4 months ago

Hello, We are trying RoRvsWild APM (currently we use Datadog) but we are facing some issues with spawning the app in production. While we understand that both solutions would conflict for obvious reasons, we don't want to disable Datadog APM during the RoRvsWild trial to avoid missing traces. I recall we had a similar issue when moving from NewRelic to Datadog years ago, but NewRelic had a setting that allowed us to use both APMs.

It seems Datadog is trying to instrument RoRvsWild plugins and we already tried to filter out the spans, but it's only applied during the post-processing step so the instrumentation will happen (or break) anyway.

Do you have any additional config/suggestion/instruction we could use to run both of them?

I'm attaching the stack trace here.

#<Thread:0x000072c87a7357e8@Datadog::Tracing::Workers::AsyncTransport <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/workers.rb:70 run> terminated with exception (report_on_exception is true):
<app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/http/instrumentation.rb:135:in `host_and_port': stack level too deep (SystemStackError)
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/http/instrumentation.rb:35:in `request'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:24:in `block in request_with_rorvswild'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/agent.rb:65:in `measure_section'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:23:in `request_with_rorvswild'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:17:in `request'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/http/instrumentation.rb:39:in `request'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:24:in `block in request_with_rorvswild'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/agent.rb:65:in `measure_section'
   ... 9218 levels...
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/workers.rb:111:in `block in perform'
  from <internal:kernel>:187:in `loop'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/workers.rb:110:in `perform'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/workers.rb:70:in `block (2 levels) in start'
bundler: failed to load command: bin/rails (bin/rails)
<app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/registry.rb:36:in `synchronize': stack level too deep (SystemStackError)
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/registry.rb:36:in `[]'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/extensions.rb:213:in `fetch_integration'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/extensions.rb:191:in `[]'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/http/instrumentation.rb:143:in `datadog_configuration'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/http/instrumentation.rb:36:in `request'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:24:in `block in request_with_rorvswild'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/agent.rb:65:in `measure_section'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:23:in `request_with_rorvswild'
   ... 9218 levels...
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/core/configuration.rb:244:in `synchronize'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/core/configuration.rb:244:in `safely_synchronize'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/core/configuration.rb:189:in `shutdown!'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/core.rb:42:in `block in <module:Datadog>'
<app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/http/instrumentation.rb:135:in `host_and_port': stack level too deep (SystemStackError)
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/http/instrumentation.rb:35:in `request'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:24:in `block in request_with_rorvswild'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/agent.rb:65:in `measure_section'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:23:in `request_with_rorvswild'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:17:in `request'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/contrib/http/instrumentation.rb:39:in `request'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/plugin/net_http.rb:24:in `block in request_with_rorvswild'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/rorvswild-1.7.0/lib/rorvswild/agent.rb:65:in `measure_section'
   ... 9218 levels...
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/workers.rb:111:in `block in perform'
  from <internal:kernel>:187:in `loop'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/workers.rb:110:in `perform'
  from <app_folder>/vendor/bundle/ruby/3.3.0/gems/ddtrace-1.22.0/lib/datadog/tracing/workers.rb:70:in `block (2 levels) in start'"
alexisbernard commented 4 months ago

The only way to instrument is by monkey patching Net::HTTP. Furthermore request calls itself when the connection is not started. That could explain the conflict. The only solution I see is to disable Net::HTTP instrumentation from either RorVsWild our DataDog:

# config/rorvswild.yml
production:
  api_key: API_KEY
  ignore_plugins:
    - NetHttp
GlauberrBatista commented 3 months ago

We were able to make it work. Thank you