[Feature] Symfony Messenger Integration

[BeyerJC]

Describe the feature you'd like

We already have an integration for symfony but it lacks the tracing for the Symfony messenger component.

I imagine it to be like the LaravelQueueIntegration and see when a worker picked up a message and all traces that happend while processing (e.g. guzzle integration may add some calls).

Is your feature request related to a problem?

No

Describe alternatives you've considered

Keep using my own solution.

Additional context

No response

[PROFeNoM]

Hello @BeyerJC 👋,

Thank you for submitting your feature request. While we have yet to estimate a time for implementing this library's instrumentation, we agree that it would be a valuable addition.

I noticed that Symfony auto-instrumentation is already available through OTel, which records spans for Symfony Messenger (open-telemetry/opentelemetry-php-contrib#215). In the meantime, you can use the OTel instrumentation with dd-trace-php support to obtain spans from this library.

Additionally, we encourage external contributions and would be happy to assist anyone with implementing this feature 😊

[natepage]

Hi @PROFeNoM 👋 ,

We recently started to use datadog to monitor our PHP applications, we are mainly using Symfony as a framework to build those.

I came across this issue after wondering where were my queue messages traces 😄 I'm completely new to dd-trace-php but gave it a go and tried to instrument Symfony Messenger, having a look at what was done for Laravel Queue.

It's far from done but I have managed to:

• instrument the message bus for both dispatching a message to the remote queue, and consuming it within the worker
• hook into the sending logic to add a custom claim onto the envelope to assure distributed tracing

I've tried to instrument the Messenger MiddlewareInterface::handle method in order to have a clear picture of all middleware executed as part of the stack within the message bus, but it's not working as expected, and I can see a span only for the very first middleware and that's it.

Here is how it looks in APM:

In order to experiment within our applications, I created a small composer package, and you can see my attempt in that instrumentation file directly: symfony-messenger-integration.php

This is very interesting and it would be of great value to us to have this integration working, and I'm happy to keep working on it, but... To be completely honest, I have no idea what I'm doing! 😄 I've managed to put this POC together by copying what other integrations are doing but some help/guidance from experts would be much appreciated! Thank you!

[PROFeNoM]

Hey @natepage :wave:

Thank you so much for your contribution!! I'll take a look a deeper look and further experiment with it ASAP after I finish wrapping up some stuff on my end 😃

I've tried to instrument the Messenger MiddlewareInterface::handle method [...] and I can see a span only for the very first middleware and that's it.

In your code, it seems that you are currently tracing this method using:

trace_method(
    'Symfony\Component\Messenger\Middleware\MiddlewareInterface',
    'handle',
    function (SpanData $span, array $args, $returnValue, $exception = null) {
        setSpanAttributes(
            $span,
            null,
            $args[0],
            $exception
        );
    }
);

However, considering that calls may be recursive, to have the behavior you describe you'd have to try the following:

trace_method(
    'Symfony\Component\Messenger\Middleware\MiddlewareInterface',
    'handle',
    [
        'posthook' => function (SpanData $span, array $args, $returnValue, $exception = null) {
            setSpanAttributes(
                $span,
                null,
                $args[0],
                $exception
            );
        },
        'recurse' => true
    ]
);

[natepage]

Hi @PROFeNoM 👋 Thank you for your quick reply!

This recurse config was exactly what I needed, thank you very much! I knew it was there but obviously didn't fully understand how it works, I've updated my code and deployed again, and it worked! 😄

[PROFeNoM]

I'm glad that it worked, @natepage!

Question: How do you feel about the volume of spans added by tracing all middleware? Do you find it noisy?

I'm asking since tracing middleware is also a recurring topic in other languages.

• For instance, you may not be interested in middleware coming from third-party libraries; only those you have the hand upon.
• Alternatively, you may only care about the first and last middleware
• [...]

What's interesting to me is how you would use this information: Do you always need it on a daily basis? Do you want this information in prod? dev? To debug/optimize?

My concerns come from the fact that due to the volume of spans added, customers could see their ingested span bytes go over the free monthly limit—I don't want customers to go over this limit for information they don't always want/need.

An alternative is adding an environment variable to decide whether middleware should be traced.

[natepage]

@PROFeNoM very good question 😄

I didn't think too much about all that to be honest, using Datadog is very new to us, especially this custom instrumentation piece so the current state of it is very much a proof of concept.

This is only my opinion on this but:

• In most cases we would care only about tracing the message handler only as it is our business logic we control and can directly work on, so it would make sense for it to be the default behaviour,
• However, having the ability to see the full picture without having to build custom instrumentation sounds amazing too as at times performance issues can be found in 3rd party dependencies and having that data would help identify these and contribute back to these potentially, so an env variable to toggle this functionality sounds good,
• We run quite a few applications, so we have created a bunch of reusable packages to implement common functionalities and some of them are providing middleware, so they are technically 3rd party dependencies, however we do "control" them, so maybe the functionality should be "Trace core middleware" this way by default it wouldn't trace middleware provided by Symfony core bundles/components/etc but trace any other middleware allowing to reduce the number of spans generated while tracing any additional middleware added from dependencies,

Again that's only me thinking out loud, and the current instrumentation code I've done is only a proof of concept. It would be great for such functionality to make it into dd-trace-php, and I'm happy to help with that so the actual PR to dd-trace-php would have an implementation that makes sense and help the most people 😄

Happy to chat further somewhere else if it makes the process easier!

[Meuk]

Hi @natepage / @PROFeNoM

I was working on exactly this to trace our message handling between services using SQS. I've instead now adopted some of the code created by @natepage for that, a big thank you for that! What we did notice that by linking the two spans, due to the distributed nature (and some slowness with consumer pods starting up automatically in our acceptance environment) is that there's sometimes a big gap between the two traces. Is there a way to prevent this, or display this differently in the DD UI?

[bwoebi]

@Meuk The only thing we currently have to offer is using Span Links instead of setting the root span parent on the other end. If you look at @natepage code, he proposes a DD_TRACE_SYMFONY_MESSENGER_DISTRIBUTED_TRACING setting for that. (Using the setting would require a small addition to the extension configuration definition - if you just want to use it, use these codeparts checking for it unconditionally.)