@decorate span() doesn't show in controller trace

[emhagman]

I have

defmodule APIWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :api
  use SpandexPhoenix
  use Sentry.Phoenix.Endpoint

At the top of my endpoint file and a route that points to my function: def theme_data(conn, params) do

In Datadog, I can see the total time the route takes but my route inside the controller has no additional spans even though I am calling functions that are decorated such as:

    # build our queries
    theme =
      API.Matches.Themes.parse_theme_from_query(params)
      |> API.Matches.Themes.add_default_jobs_if_empty(user_id, group_id)

In the route where add_default_jobs_if_empty is decorated like so:

@decorate span()
def add_default_jobs_if_empty(parsed_theme, user_id, group_id) do
  jobs = parsed_theme["jobs"]
  job_ids = if !jobs, do: open_job_ids_for_group(user_id, group_id), else: jobs
  Map.put(parsed_theme, "jobs", job_ids)
end

Any thoughts???

[dnlserrano]

Hey @emhagman, sorry I only got to answer now. I am not an official core team member (@zachdaniel and @GregMefford can help better). But since I helped reinstate the decorators and we are using it in Production successfully at Onfido, I thought I might try and help.

Whenever I don't see traces for some reason, I run the Datadog agent itself or (for maybe a quicker spin) I run my little hacktracing project that prints out the traces it is receiving (as if it were the Datadog agent, also running in port 8126).

If you do this, i.e., point to either Datadog agent or hacktracing, can you see the traces being printed in that service? If so, that should mean the traces are being propagated correctly. If not, we can investigate further. 🤓 The logging can also allow you to understand if you're loosing traces because you're wrongfully generating a new trace ID (which would cause the trace to be different in Datadog).

Let me know. And thanks for giving spandex a go! 😄

[GregMefford]

Any news on this issue, @emhagman?

Something to check if you didn't already - you have to configure the default tracer to use with Decorators:

# config/config.exs

use Mix.Config

config :spandex, :decorators, tracer: MyApp.Tracer

[emhagman]

@dnlserrano Thanks, that's very helpful, I'll try it out.

@GregMefford I actually had two separate tracers going for separate apps and I had config :spandex, :decorators, tracer: Graph.Tracer set for one app but was missing my API tracer config :spandex, :decorators, tracer: API.Tracer. Unfortunately, it doesn't appear to be working still.

I'll keep you posted.

[GregMefford]

Ah ok. So the problem might be that configuration is global in an umbrella application, if that's what you're talking about, so only the last of those settings to get applied will have any effect, since they're both configuring the :spandex, :decorators, :tracer config key.

Unfortunately, the only way around that would be to pick one (or make a third Tracer for this purpose) and override settings as needed in the decorator calls: @decorate span(service: :graph), etc.

[zachdaniel]

Ah, yeah and here we see why we should have defined the decorators inside each tracer, so you could say use MyApp.Tracer.Decorators.

[GregMefford]

Yeah, the way I've been using this is in the case where each service is deployed independently, so we essentially always have one tracer per GitHub repo / deployment and it just works out. I realize that's not how everyone does their apps though. We'll take the learning for when we end up building something new for OpenTelemetry. 👍

[emhagman]

Thanks for the comments guys, makes sense. I'll use one tracer globally and get back to you :)

[emhagman]

😢It's so beautiful... it works 😁 Thanks!!!

[zachdaniel]

Looks like this is resolved.