Only leaf spans

[ludwikbukowski]

I have the problem so that I'd like to include only bottom level spans in my Trace (only leaf spans) So that starting from some distinguished span, only the spans that does not have childs will be forwarded. Is it possible to achieve that without creating the fork of this library and implementing such a feature?

[GregMefford]

Can you clarify what you're trying to accomplish / why you need this feature? It sounds like you'd prefer for each trace to only display the root span (which is required as far as I know) and the most-detailed spans, but nothing in between (i.e. no intermediate calls in the flame graph).

Here are a few thoughts while we wait for more details:

• Spandex itself doesn't create any traces or spans unless you tell it to. If you don't create the intermediate spans in the first place, they won't be there. I presume what you're looking for, though, is more control over the spans that get automatically created e.g. by spandex_phoenix and spandex_ecto.

• You could create your own sender to plug into spandex_datadog that filters some of the spans out of the trace before sending it. It's not obvious in the docs, but then if you configure the sender: MyApp.SpecialDatadogSender option in your Tracer config, it will call that module instead of SpandexDatadog.APIServer. You could make a module that uses defdelegate to call into the real SpandexDatadog.APIServer for everything except the send_trace function. (see https://github.com/spandex-project/spandex_datadog/blob/b2dd1c0/lib/spandex_datadog/api_server.ex#L105 for how this works normally).

• If it turns out that what you want is something that others would also want, we can figure out whether we should / how we can integrate it into Spandex or SpandexDatadog itself.

[ludwikbukowski]

Thanks for clear and very good detailed response :) Exactly this is what I would like to have. The reason is that we wrote support for Absinthe library so that each resolver is span and the Trace is simply phoenix request. It was done by implementing custom middleware that calls start_span for each field (btw maybe I could share that code with you as something like spandex_absinthe)

The issue we faced is that we have quite deep flame graph trees. The resolvers that calls other resolvers aren't very informative for us since those that do some time-consuming operations are the bottom level resolvers only.

I will look at the code you suggest and try to figure out optimal solution that does not require modifying the library itself :)

[GregMefford]

Cool! Yes we would love to have a spandex_absinthe library that people can just drop into their project. Maybe that library could have some way to “compress” the stack of spans as well. I’m not sure how Absinthe middleware works.

[zachdaniel]

Agreed! Greg hit it all on the head. And absinthe is definitely something that deserves its own full integration :)

[zachdaniel]

I think we should close this now, as there isn't really much sense in giving spandex an option to only send leaf traces, but rather in building an integration with absinthe separately.

[ludwikbukowski]

Hey! Let's suppose I'd like to contribute regarding the Absinthe support Should I create PR into root spandex library so that you can review and in the end create separate spandex_absinthe repository?

[zachdaniel]

I would recommend creating your own repository, and making changes there. Then we will review your repo, and you can transfer it to the spandex-project org. I would also put you on the core team so you could help maintain it :)

[ludwikbukowski]

Oh right, makes much more sense :) Then will keep you updated!

[cheerfulstoic]

Hey @ludwikbukowski Do you still have that middleware for spandex + absinthe? I'm struggling a bit trying to get it to work. I followed this presentation (slide 37 is the main one for this) and I got spans in my Datadog traces, but they were more of a single, long spike of spans going down getting smaller and smaller, rather than the flamegraph that I'm used to.

The one thing I would point out about that slide is that it shows something like this:

  def middleware(middleware, _field, _object) do
    [MyAppWeb.Graphql.Middleware.DatadogTracing | middleware]

    middleware
  end

But that wouldn't actually add the middleware. It only "works" (partly) when I do this:

  def middleware(middleware, _field, _object) do
    [MyAppWeb.Graphql.Middleware.DatadogTracing | middleware]
  end

[cheerfulstoic]

Oh, wow, ok, I see that's your slideshow from after you made your comments 😆

Maybe there's more to it? This is what my middleware module looks like:

defmodule MyAppWeb.Graphql.Middleware.DatadogTracing do
  @moduledoc "Allows display on a flamegraph for GraphQL fields in Datadog traces"

  @behaviour Absinthe.Middleware

  alias MyAppWeb.Tracer

  @impl Absinthe.Middleware
  def call(res, _config) do
    Tracer.start_span("#{res.parent_type.name}.#{res.definition.name}")
    %{res | middleware: res.middleware ++ [{{__MODULE__, :after_field}, []}]}
  end

  def after_field(res, _) do
    Tracer.finish_span()

    res
  end
end

And this is an example of what I see in Datadog:

[abmBispo]

Any updates regarding the spandex_absinthe? The code of @cheerfulstoic is great to get the flamegraphs for the absinthe resolvers, however it doesn't help in segregating the requests. Any recommendations to break apart these distinct queries, like we have in normal REST requests?

[abmBispo]

To whom it may concern, I made this gambiarra so now I can convert the normal resource (HTTP Verb + Path) into the query/mutation name.

MyApp.Tracer

defmodule MyApp.Tracer do
  @moduledoc false
  use Spandex.Tracer, otp_app: :my_app

  def customize_metadata(%{body_params: %{"query" => query}} = conn) when is_binary(query) do
    name =
      query
      |> String.split("{")
      |> Enum.at(1)
      |> String.split("(")
      |> Enum.at(0)
      |> String.replace("\n", " ")
      |> String.replace(" ", "")

    conn
    |> SpandexPhoenix.default_metadata()
    |> Keyword.update(:resource, name, fn _ -> name end)
    |> Keyword.merge(tags: [query: conn.body_params["query"], variables: conn.body_params["variables"]])
  end

  def customize_metadata(conn),
    do: SpandexPhoenix.default_metadata(conn)
end

Application

defmodule MyApp.Application do
  ...
  SpandexPhoenix.Telemetry.install(customize_metadata: &MyApp.Tracer.customize_metadata/1)
  ...
end

[LucasAmaral42]

I did something like you (adaptação técnica avançada/gambiarra), but using :before_send.

router.ex

post "/graphql", Absinthe.Plug,
  schema: MyAppWeb.Schema,
  before_send: {__MODULE__, :absinthe_before_send}

...

def absinthe_before_send(conn, %Absinthe.Blueprint{result: %{data: data}} = blueprint) do
    with [schema | rest] <- Map.keys(data),
         [] <- rest,
         type <- Absinthe.Blueprint.current_operation(blueprint).type do
        conn
        |> put_private(:"graphql-schema", "#{type} #{schema}")
    else
        _ -> conn
    end
end

Keeping the POST / on cases when is more than 1 query/mutation

MyApp.Tracer

defmodule MyApp.Tracer do
  @moduledoc "tracer information to datadog"

  use Spandex.Tracer, otp_app: :my_app

  def customize_metadata(%Plug.Conn{private: %{:"graphql-schema" => resource}} = conn) do
    conn
    |> SpandexPhoenix.default_metadata()
    |> Keyword.update(:resource, "", fn default_resource ->
      resource || default_resource
    end)
  end

  def customize_metadata(conn), do: SpandexPhoenix.default_metadata(conn)
end

Creating a customize_metadata to update resource that is sent.

application.ex

SpandexPhoenix.Telemetry.install(
      span_opts: [
        type: :web,
        tags: [version: version]
      ],
      customize_metadata: &MyApp.Tracer.customize_metadata/1
    )

Adding the customize_metadata to SpandexPhoenix.Telemetry.install opts.

---

On that way only updates resource in single query/mutation/subscription graphql requests and keep the others requests as usual.