How do I totally turn off tracing in local development? Why is datadog still trying to make network calls?

[alexevanczuk]

When I run my Ruby tests, I am getting this:

E, [2023-02-21T00:00:01.175004 #34286] ERROR -- ddtrace: [ddtrace] (/Users/alexevanczuk/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/ddtrace-1.23.3/lib/datadog/tracing/transport/http/client.rb:41:in `rescue in send_request') Internal error during Datadog::Tracing::Transport::HTTP::Client request. Cause: Errno::ECONNREFUSED Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126) Location: /Users/alexevanczuk/.rbenv/versions/3.3.5/lib/ruby/3.3.0/net/http.rb:1603:in `initialize'

However, I have set this configuration:

require 'ddtrace/auto_instrument'
require 'version_tracking'
require 'datadog/appsec'

is_test = ENV['CI'] || Rails.env.test?
Datadog.configure do |c|
  if !is_test
    # ... sets up tracing
  elsif is_test
    c.tracing.test_mode.enabled = true
  end

  c.env = Rails.env
  c.service = DatadogHelper.service_name
  c.version = git_sha

  c.tags = {...}
end

Thank you for your help!

[Decker87]

+1 on this request... below is a screenshot showing how overwhelming the datadog output can be.

[alexevanczuk]

I noticed in this file lib/datadog/core/diagnostics/environment_logger.rb there is this code snippet:

module Datadog
  module Core
    module Diagnostics
      # Base class for EnvironmentLoggers - should allow for easy reporting by users to Datadog support.
      #
      # The EnvironmentLogger should not pollute the logs in a development environment.
....
        # Are we logging the environment data?
        def log?
          startup_logs_enabled = Datadog.configuration.diagnostics.startup_logs.enabled
          if startup_logs_enabled.nil?
            # Do not pollute the logs in a development environment.
            !Datadog::Core::Environment::Execution.development?
          else
            startup_logs_enabled
          end
        end
      end

I also see a setting:

diagnostics.startup_logs.enabled

However, despite setting this, I'm still seeing the startup logs in development 🤔

[ivoanjo]

Hey folks! As usual, thanks for reaching out, and sorry for the annoyance.

TL;DR: Setting the environment variables DD_TRACE_ENABLED=false and DD_TRACE_STARTUP_LOGS=false should get you a clean run without dd-trace-rb logging anything.

I answer in more detail below ;)

Hopefully this helps, and LMK if it works for you.

---

The logs you shared aren't all coming from the same place so let me break down my answer into a few parts:

1. Error when trying to report data while in test_mode

E, [2023-02-21T00:00:01.175004 #34286] ERROR -- ddtrace: [ddtrace] (/Users/alexevanczuk/.rbenv/versions/3.3.5/lib/ruby/gems/3.3.0/gems/ddtrace-1.23.3/lib/datadog/tracing/transport/http/client.rb:41:in rescue in send_request') Internal error during Datadog::Tracing::Transport::HTTP::Client request. Cause: Errno::ECONNREFUSED Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126) Location: /Users/alexevanczuk/.rbenv/versions/3.3.5/lib/ruby/3.3.0/net/http.rb:1603:ininitialize'

From what you shared, it looks like you may want to disable tracing instead of setting test_mode, e.g. something like

Datadog.configure do |c|
  if !is_test
    # ... sets up tracing
  elsif is_test
    c.tracing.enabled = false
  end
  # ... 
end

(also available via DD_TRACE_ENABLED env variable).

The (confusingly named) test_mode is for still collecting traces, e.g. when testing integrations or dd-trace-rb. To be very fair, I've spotted that our documentation states

When test mode is enabled, the tracer uses a Test adapter for no-op transport

which has not been true for a long time. I'll raise this with the team to correct the docs.

2. Noisy DATADOG CONFIGURATION logs

The DATADOG CONFIGURATION logs can indeed be disabled with the diagnostics.startup_logs.enabled setting, but there's a catch sometimes (see below).

We've reduced the verbosity of logs by default in #3785 (I chased that one!), since yeah it was kinda crazy in some cases. Unfortunately, since it's not a security or a bugfix, it's probably not going to get backported to the 1.x series :(

3. Still getting CONFIGURATION logs with diagnostics.startup_logs.enabled = false

This is a bit of the sharper edge with how dd-trace-rb works right now. TL;DR require 'datadog/auto_instrument' triggers dd-trace-rb initialization eagerly, which means the logs get printed before the Datadog.configure gets executed and turns them off.

As a workaround, I recommend setting it via environment variable, e.g. DD_TRACE_STARTUP_LOGS=false, since this will be picked up from the get-go.

[alexevanczuk]

Thank you so much for your help @ivoanjo . I've followed these instructions and an initial test suggests this seemed to eliminate the logs!

I also appreciate you following up on getting the documentation to match the current behavior!

I've started the upgrade to 2.0... I didn't even realize we were able to upgrade today. I see there's an upgrade guide, but do you have any reading on what the upgrade gets us behaviorally? I'll check out the CHANGELOG but just wondering if there's a more holistic pitch somewhere on 2.0

[ivoanjo]

I've started the upgrade to 2.0... I didn't even realize we were able to upgrade today. I see there's an upgrade guide, but do you have any reading on what the upgrade gets us behaviorally? I'll check out the CHANGELOG but just wondering if there's a more holistic pitch somewhere on 2.0

Compared to 1.x, the 2.x series has a lot of improvements to the profiler and appsec features, and a bunch of bugfixes all around (since only high-priority stuff is getting backported to 1.x).

So if you're using appsec and profiling you should get some nice shiny features; otherwise you're mostly getting a bit more bugfixes and polish ;)

The upgrade guide, since we tried to be extensive, may look a bit more scary than it actually is. The 1.x to 2.x jump was mostly so we could drop support for old Rubies + move a few constants around and clean up some old methods. Those changes are backwards-incompatible, hence the bump to the major version.

Feel free to reach out via here and/or support if you do run into issues/have any questions :)

[alexevanczuk]

Amazing! If it's okay I'll close this for now then and will ping ya if anything comes up. Thanks again :)