search

DataDog / dd-trace-rb

Datadog Tracing Ruby Client
https://docs.datadoghq.com/tracing/
Other
310 stars 375 forks source link

ddtrace running without Datadog configured #1437

Closed jshah closed 3 years ago

jshah commented 3 years ago

We're using ddtrace 0.45.0 and we're controlling tracer as such:

if ENV['ENABLE_DATADOG'] && Rails.env.production? && !defined?(Rails::Console)
    Datadog.configure do |c|
      c.tracer.enabled = true
      c.tracer hostname: open('http://169.254.169.254/latest/meta-data/local-ipv4',
                              { read_timeout: 3, open_timeout: 3 }).read
    end
else
  # We only want ddtrace to be run in production environments.
  # Tracer is on by default so have to explicitly turn it off.
  Datadog.configure do |c|
    c.tracer.enabled = false
  end
end

We also use the https://github.com/grosser/parallel_tests gem and when running RAILS_ENV=test bundle exec rails parallel:setup, I always see this output from ddtrace:

E, [2021-03-30T17:59:00.780056 #248] ERROR -- ddtrace: [ddtrace] (/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-0.45.0/lib/ddtrace/transport/http/client.rb:35:in `rescue in send_request') Internal error during HTTP transport request. Cause: Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126) Location: /usr/local/lib/ruby/2.6.0/net/http.rb:949:in `rescue in block in connect'
457 | I, [2021-03-30T17:59:00.783870 #248]  INFO -- ddtrace: [ddtrace] DATADOG TRACER CONFIGURATION - {"date":"2021-03-30T17:59:00+00:00","os_name":"x86_64-pc-linux-gnu","version":"0.45.0","lang":"ruby","lang_version":"2.6.6","enabled":true,"agent_url":"http://127.0.0.1:8126?timeout=1","agent_error":"Datadog::Transport::InternalErrorResponse ok?: unsupported?:, not_found?:, client_error?:, server_error?:, internal_error?:true, payload:, error_type:Errno::ECONNREFUSED error:Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for \"127.0.0.1\" port 8126)","debug":false,"analytics_enabled":false,"runtime_metrics_enabled":false,"integrations_loaded":"active_model_serializers@,aws@3.90.1,concurrent_ruby@1.1.8,dalli@,delayed_job@4.1.5,elasticsearch@,ethon@,excon@0.71.1,faraday@0.15.4,grape@,graphql@,grpc@,http@2.6.6,httpclient@2.8.3,httprb@,kafka@,presto@,que@,mysql2@,mongo@,qless@,racecar@,rails@5.2.4.5,rake@12.3.3,redis@4.0.3,resque@,rest_client@2.0.2,sequel@5.28.0,shoryuken@,sidekiq@5.2.2,sinatra@2.0.8.1,sneakers@,sucker_punch@","vm":"ruby-2.6.6","partial_flushing_enabled":false,"priority_sampling_enabled":false,"health_metrics_enabled":false,"integration_active_model_serializers_analytics_enabled":"false","integration_active_model_serializers_analytics_sample_rate":"1.0","integration_active_model_serializers_enabled":"true","integration_active_model_serializers_service_name":"active_model_serializers","integration_aws_analytics_enabled":"false","integration_aws_analytics_sample_rate":"1.0","integration_aws_enabled":"true","integration_aws_service_name":"aws","integration_concurrent_ruby_analytics_enabled":"false","integration_concurrent_ruby_analytics_sample_rate":"1.0","integration_concurrent_ruby_enabled":"true","integration_concurrent_ruby_service_name":"concurrent-ruby","integration_dalli_analytics_enabled":"false","integration_dalli_analytics_sample_rate":"1.0","integration_dalli_enabled":"true","integration_dalli_service_name":"memcached","integration_delayed_job_analytics_enabled":"false","integration_delayed_job_analytics_sample_rate":"1.0","integration_delayed_job_enabled":"true","integration_delayed_job_service_name":"delayed_job","integration_delayed_job_client_service_name":"delayed_job-client","integration_delayed_job_error_handler":"#\u003cProc:0x000055e558f3b090@/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-0.45.0/lib/ddtrace/tracer.rb:28\u003e","integration_elasticsearch_analytics_enabled":"false","integration_elasticsearch_analytics_sample_rate":"1.0","integration_elasticsearch_enabled":"true","integration_elasticsearch_service_name":"elasticsearch","integration_elasticsearch_quantize":"{}","integration_ethon_analytics_enabled":"false","integration_ethon_analytics_sample_rate":"1.0","integration_ethon_enabled":"true","integration_ethon_service_name":"ethon","integration_ethon_distributed_tracing":"true","integration_ethon_split_by_domain":"false","integration_excon_analytics_enabled":"false","integration_excon_analytics_sample_rate":"1.0","integration_excon_enabled":"true","integration_excon_service_name":"excon","integration_excon_distributed_tracing":"true","integration_excon_error_handler":"","integration_excon_split_by_domain":"false","integration_faraday_analytics_enabled":"false","integration_faraday_analytics_sample_rate":"1.0","integration_faraday_enabled":"true","integration_faraday_service_name":"faraday","integration_faraday_distributed_tracing":"true","integration_faraday_error_handler":"#\u003cProc:0x000055e55977e248@/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-0.45.0/lib/ddtrace/contrib/faraday/configuration/settings.rb:11 (lambda)\u003e","integration_faraday_split_by_domain":"false","integration_grape_analytics_enabled":"","integration_grape_analytics_sample_rate":"1.0","integration_grape_enabled":"true","integration_grape_service_name":"grape","integration_grape_error_statuses":"","integration_graphql_analytics_enabled":"","integration_graphql_analytics_sample_rate":"1.0","integration_graphql_enabled":"true","integration_graphql_service_name":"ruby-graphql","integration_graphql_schemas":"","integration_grpc_analytics_enabled":"false","integration_grpc_analytics_sample_rate":"1.0","integration_grpc_enabled":"true","integration_grpc_service_name":"grpc","integration_http_analytics_enabled":"false","integration_http_analytics_sample_rate":"1.0","integration_http_enabled":"true","integration_http_service_name":"net/http","integration_http_distributed_tracing":"true","integration_http_split_by_domain":"false","integration_httpclient_analytics_enabled":"false","integration_httpclient_analytics_sample_rate":"1.0","integration_httpclient_enabled":"true","integration_httpclient_service_name":"httpclient","integration_httpclient_distributed_tracing":"true","integration_httpclient_split_by_domain":"false","integration_httprb_analytics_enabled":"false","integration_httprb_analytics_sample_rate":"1.0","integration_httprb_enabled":"true","integration_httprb_service_name":"httprb","integration_httprb_distributed_tracing":"true","integration_httprb_split_by_domain":"false","integration_kafka_analytics_enabled":"false","integration_kafka_analytics_sample_rate":"1.0","integration_kafka_enabled":"true","integration_kafka_service_name":"kafka","integration_presto_analytics_enabled":"false","integration_presto_analytics_sample_rate":"1.0","integration_presto_enabled":"true","integration_presto_service_name":"presto","integration_que_analytics_enabled":"false","integration_que_analytics_sample_rate":"1.0","integration_que_enabled":"true","integration_que_service_name":"que","integration_que_distributed_tracing":"true","integration_que_tag_args":"false","integration_que_tag_data":"false","integration_que_error_handler":"#\u003cProc:0x000055e558f3b090@/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-0.45.0/lib/ddtrace/tracer.rb:28\u003e","integration_mysql2_analytics_enabled":"false","integration_mysql2_analytics_sample_rate":"1.0","integration_mysql2_enabled":"true","integration_mysql2_service_name":"mysql2","integration_mongo_analytics_enabled":"false","integration_mongo_analytics_sample_rate":"1.0","integration_mongo_enabled":"true","integration_mongo_service_name":"mongodb","integration_mongo_quantize":"{:show=\u003e[:collection, :database, :operation]}","integration_qless_analytics_enabled":"false","integration_qless_analytics_sample_rate":"1.0","integration_qless_enabled":"true","integration_qless_service_name":"qless","integration_qless_tag_job_data":"false","integration_qless_tag_job_tags":"false","integration_racecar_analytics_enabled":"false","integration_racecar_analytics_sample_rate":"1.0","integration_racecar_enabled":"true","integration_racecar_service_name":"racecar","integration_rails_analytics_enabled":"","integration_rails_analytics_sample_rate":"1.0","integration_rails_enabled":"true","integration_rails_service_name":"","integration_rails_cache_service":"","integration_rails_controller_service":"","integration_rails_database_service":"","integration_rails_distributed_tracing":"true","integration_rails_exception_controller":"","integration_rails_middleware":"true","integration_rails_middleware_names":"false","integration_rails_template_base_path":"views/","integration_rails_log_injection":"false","integration_rake_analytics_enabled":"false","integration_rake_analytics_sample_rate":"1.0","integration_rake_enabled":"true","integration_rake_service_name":"rake","integration_rake_quantize":"{}","integration_redis_analytics_enabled":"false","integration_redis_analytics_sample_rate":"1.0","integration_redis_enabled":"true","integration_redis_service_name":"redis","integration_redis_command_args":"true","integration_resque_analytics_enabled":"false","integration_resque_analytics_sample_rate":"1.0","integration_resque_enabled":"true","integration_resque_service_name":"resque","integration_resque_workers":"[]","integration_resque_error_handler":"#\u003cProc:0x000055e558f3b090@/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-0.45.0/lib/ddtrace/tracer.rb:28\u003e","integration_rest_client_analytics_enabled":"false","integration_rest_client_analytics_sample_rate":"1.0","integration_rest_client_enabled":"true","integration_rest_client_service_name":"rest_client","integration_rest_client_distributed_tracing":"true","integration_sequel_analytics_enabled":"false","integration_sequel_analytics_sample_rate":"1.0","integration_sequel_enabled":"true","integration_sequel_service_name":"","integration_shoryuken_analytics_enabled":"false","integration_shoryuken_analytics_sample_rate":"1.0","integration_shoryuken_enabled":"true","integration_shoryuken_service_name":"shoryuken","integration_shoryuken_error_handler":"#\u003cProc:0x000055e558f3b090@/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-0.45.0/lib/ddtrace/tracer.rb:28\u003e","integration_sidekiq_analytics_enabled":"false","integration_sidekiq_analytics_sample_rate":"1.0","integration_sidekiq_enabled":"true","integration_sidekiq_service_name":"sidekiq","integration_sidekiq_tag_args":"false","integration_sidekiq_client_service_name":"sidekiq-client","integration_sidekiq_error_handler":"#\u003cProc:0x000055e558f3b090@/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-0.45.0/lib/ddtrace/tracer.rb:28\u003e","integration_sinatra_analytics_enabled":"","integration_sinatra_analytics_sample_rate":"1.0","integration_sinatra_enabled":"true","integration_sinatra_service_name":"sinatra","integration_sinatra_distributed_tracing":"true","integration_sinatra_headers":"{:response=\u003e[\"Content-Type\", \"X-Request-ID\"]}","integration_sinatra_resource_script_names":"false","integration_sneakers_analytics_enabled":"false","integration_sneakers_analytics_sample_rate":"1.0","integration_sneakers_enabled":"true","integration_sneakers_service_name":"sneakers","integration_sneakers_error_handler":"#\u003cProc:0x000055e558f3b090@/app/vendor/bundle/ruby/2.6.0/gems/ddtrace-0.45.0/lib/ddtrace/tracer.rb:28\u003e","integration_sneakers_tag_body":"false","integration_sucker_punch_analytics_enabled":"false","integration_sucker_punch_analytics_sample_rate":"1.0","integration_sucker_punch_enabled":"true","integration_sucker_punch_service_name":"sucker_punch"}
458 | W, [2021-03-30T17:59:00.783928 #248]  WARN -- ddtrace: [ddtrace] DATADOG TRACER DIAGNOSTIC - Agent Error: Datadog::Transport::InternalErrorResponse ok?: unsupported?:, not_found?:, client_error?:, server_error?:, internal_error?:true, payload:, error_type:Errno::ECONNREFUSED error:Failed to open TCP connection to 127.0.0.1:8126 (Connection refused - connect(2) for "127.0.0.1" port 8126)
459

I'm not sure if this is specific to the gem but it wasn't happening before ddtrace was installed

marcotc commented 3 years ago

👋 @jshah, thank you for this report. Could you let us know how you are including/requiring the ddtrace in your project? (the relevant Gemfile line and require statement) This will help us guide you on how to avoid these messages in development, as the way you require the gem affects how it starts up.

jshah commented 3 years ago

Hi @marcotc!

We are requiring the gem as following in no specific group. Should we be requiring only in the :production group?

gem "ddtrace", "~> 0.45.0", require: 'ddtrace/auto_instrument'
jshah commented 3 years ago

Ah, I just grasped that auto_instrument will automatically turn on tracing. Do you know what services auto_instrument runs by default?

marcotc commented 3 years ago

auto_instrument will instrument every gem available in your application. It will also kick off all tracer internals, including the part that prints ddtrace: [ddtrace] DATADOG TRACER....

I suggest removing , require: 'ddtrace/auto_instrument' from your Gemfile and adding that require statement here:

if ENV['ENABLE_DATADOG'] && Rails.env.production? && !defined?(Rails::Console)
    require: 'ddtrace/auto_instrument'

    Datadog.configure do |c|
      c.tracer hostname: open('http://169.254.169.254/latest/meta-data/local-ipv4',
                              { read_timeout: 3, open_timeout: 3 }).read
    end
end

Unless you are directly interacting with the gem (e.g. making calls to Datadog. ...), you can even do away with the else block above. If you are directly interacting with it, then add require 'ddtrace' to the else block. This won't do anything, except for loading classes.

jshah commented 3 years ago

Thanks, this makes sense. Closing issue 🙂