Can't get data sent to datadog

[michaeldegli]

Hi,

I'm working with this library but I can't get information sent to datadog. Please tell me if there is any other diagnostic information I can provide or troubleshooting steps I've missed. I've set up the application as the README suggests and then when my application is running where the datadog agent is, I run the following commands in iex:

Tracer.start_trace("foo")
Tracer.update_span(service: :my_service, type: :web, resource: "/bar")
Tracer.finish_span()
Tracer.finish_trace()

What else I've tried

• sending a sample request using curl on the same host using the datadog API guide (this works)

Source Files:

• Application: https://gist.github.com/michaeldegli/c6f63587cf3718ba7bdfdbc558fdfbc7
• Module: https://gist.github.com/michaeldegli/f251cb3edb72cd4450245056bdb7fd4d

Thank you

[dnlserrano]

Hey @michaeldegli I can maybe try and help.

Have you tried checking if the trace is actually being created and finished as you'd expect? E.g.:

started_trace = Tracer.start_trace("foo")
IO.inspect(started_trace, label: "started_trace")

updated_span = Tracer.update_span(service: :my_service, type: :web, resource: "/bar")
IO.inspect(updated_span, label: "updated_span")

finished_span = Tracer.finish_span()
IO.inspect(finished_span, label: "finished_span")

finished_trace = Tracer.finish_trace()
IO.inspect(finished_trace, label: "finished_trace")

Can you past the outputs of those?

Also, can you add this function to your tracer, please to get the outputs of your config:

defmodule Tracer do
  @otp_app :tracer

  use Spandex.Tracer, otp_app: @otp_app

  defp get_config() do
    config([], @otp_app)
  end
end

Call it simply with: Tracer.get_config().

[michaeldegli]

Hi, @dnlserrano here is the output.

started_trace: {:ok,
 %Spandex.Trace{
   baggage: [],
   id: 5705550572390978275,
   priority: 1,
   spans: [],
   stack: [
     %Spandex.Span{
       completion_time: nil,
       env: "dev",
       error: nil,
       http: nil,
       id: 3466231812659300695,
       name: "foo",
       parent_id: nil,
       private: [],
       resource: nil,
       service: :funnelcloud,
       services: [],
       sql_query: nil,
       start: 1558965140268412171,
       tags: [],
       trace_id: 5705550572390978275,
       type: nil
     }
   ]
 }}
{:ok,
 %Spandex.Trace{
   baggage: [],
   id: 5705550572390978275,
   priority: 1,
   spans: [],
   stack: [
     %Spandex.Span{
       completion_time: nil,
       env: "dev",
       error: nil,
       http: nil,
       id: 3466231812659300695,
       name: "foo",
       parent_id: nil,
       private: [],
       resource: nil,
       service: :funnelcloud,
       services: [],
       sql_query: nil,
       start: 1558965140268412171,
       tags: [],
       trace_id: 5705550572390978275,
       type: nil
     }
   ]
 }}

 updated_span: {:ok,
 %Spandex.Span{
   completion_time: nil,
   env: "dev",
   error: nil,
   http: nil,
   id: 3466231812659300695,
   name: "foo",
   parent_id: nil,
   private: [],
   resource: "/bar",
   service: :my_service,
   services: [],
   sql_query: nil,
   start: 1558965140268412171,
   tags: [],
   trace_id: 5705550572390978275,
   type: :web
 }}
{:ok,
 %Spandex.Span{
   completion_time: nil,
   env: "dev",
   error: nil,
   http: nil,
   id: 3466231812659300695,
   name: "foo",
   parent_id: nil,
   private: [],
   resource: "/bar",
   service: :my_service,
   services: [],
   sql_query: nil,
   start: 1558965140268412171,
   tags: [],
   trace_id: 5705550572390978275,
   type: :web
 }}

 finished_span: {:ok,
 %Spandex.Span{
   completion_time: 1558965239511254550,
   env: "dev",
   error: nil,
   http: nil,
   id: 3466231812659300695,
   name: "foo",
   parent_id: nil,
   private: [],
   resource: "/bar",
   service: :my_service,
   services: [],
   sql_query: nil,
   start: 1558965140268412171,
   tags: [],
   trace_id: 5705550572390978275,
   type: :web
 }}
{:ok,
 %Spandex.Span{
   completion_time: 1558965239511254550,
   env: "dev",
   error: nil,
   http: nil,
   id: 3466231812659300695,
   name: "foo",
   parent_id: nil,
   private: [],
   resource: "/bar",
   service: :my_service,
   services: [],
   sql_query: nil,
   start: 1558965140268412171,
   tags: [],
   trace_id: 5705550572390978275,
   type: :web
 }}

 finished_trace: {:ok,
 %Spandex.Trace{
   baggage: [],
   id: 5705550572390978275,
   priority: 1,
   spans: [
     %Spandex.Span{
       completion_time: 1558965239511254550,
       env: "dev",
       error: nil,
       http: nil,
       id: 3466231812659300695,
       name: "foo",
       parent_id: nil,
       private: [],
       resource: "/bar",
       service: :my_service,
       services: [],
       sql_query: nil,
       start: 1558965140268412171,
       tags: [],
       trace_id: 5705550572390978275,
       type: :web
     }
   ],
   stack: []
 }}
{:ok,
 %Spandex.Trace{
   baggage: [],
   id: 5705550572390978275,
   priority: 1,
   spans: [
     %Spandex.Span{
       completion_time: 1558965239511254550,
       env: "dev",
       error: nil,
       http: nil,
       id: 3466231812659300695,
       name: "foo",
       parent_id: nil,
       private: [],
       resource: "/bar",
       service: :my_service,
       services: [],
       sql_query: nil,
       start: 1558965140268412171,
       tags: [],
       trace_id: 5705550572390978275,
       type: :web
     }
   ],
   stack: []
 }}

Tracer.get_config()

[
  trace_key: Tracer,
  tags: [],
  services: [],
  private: [],
  strategy: Spandex.Strategy.Pdict,
  service: :funnelcloud,
  adapter: SpandexDatadog.Adapter,
  disabled?: false,
  env: "dev"
]

Mix.env() outputs :dev

[dnlserrano]

Everything seems OK. Can you try setting the sync_threshold to 10 and batch_size options to 1 and try again? You might be batching requests and those might only reach the Datadog client after a while (hence batch_size to 1, to not batch). sync_threshold to 10 is just to guarantee no funky stuff is going on regarding sync/async work (everything should be async with 10). More info here.

Let me know.

[zachdaniel]

Hey @michaeldegli, sorry for my radio silence the past couple days. Have you looked at the logs for the datadog agent itself? This might be an issue w/ the configuration of your datadog agent. For instance, I don't believe that most versions of the agent default to having APM enabled.

[michaeldegli]

Hi @zachdaniel, looks like things are working now that we're not batching traces with low volume testing. I Appreciate your help @dnlserrano. 👍

[zachdaniel]

Awesome! Thanks for taking over on this @dnlserrano!

[Xosmond]

This also helped me! Thanks!