Tracing with async/CLI setups

[SonarSoftware]

I'm currently using Swoole, but I'm guessing it's a similar issue for ReactPHP. Is there any direction on getting it working on an environment like this today? Or any plans to implement?

[labbati]

Hi @SonarSoftware, we definitely have plans for ReactPHP as it came up a few times now, and I guess a similar solution might apply to Swoole as well, as you are suggesting.

I cannot provide an ETA at the moment, though. Given that I am not expert about RactPHP/Swoole internals (I only can assume that there is some separate thread/event loop under the hood) this would make it difficult for you to support tracing without some work on our side due to the following reasons that comes to my mind (@SammyK and @morrisonlevi feel free to jump in as you might have more experience than me with those libraries):

• at the moment we flush traces on a register_shutdown_function. This would not work in your case as your script (I assume) runs as long as the app runs.
• Even if you manual invoke GlobalTracer::get()->flush() in some after-response hook that is provided by Swoole/ReactPHP, the Tracer remains in an unpredictable state as it approaches serving next request.
• On top of that, and this might be the really tricky part, We use at the moment one global tracer instance (as per OpenTracing approach) and span coming from different requests I am pretty sure would end up mixed in a giant basket.

Again, I have never used such libraries so I can only talk from the top of my head.

[SonarSoftware]

Yes, it is tricky. There are some ways around the global tracer issue - Swoole for example has a sandbox that is used to separate per-request items out. That being said, it's still not easy and it would definitely take some work.

It might be worth taking a look at https://github.com/upscalesoftware/swoole-newrelic for some inspiration - I've been poking at it a bit seeing if we can do something similar with ddtrace.

[nesl247]

Any update on this? I ask because we are in the process of migrating from NewRelic and ran into this. It is not an issue we expected to show up.

[morrisonlevi]

I am not aware of any updates specific to ReactPHP or Swoole, but I know some of the underlying engineering challenges that assume a single global trace are being worked on.

One thing to set expectations on is what in these products gets traced. To that end you could help by sharing what features you are using. As an example, I found a React\Http\Server class -- I assume that users would want each new request that the server processes to get traced independently. Telling us which pieces of Swoole and ReactPHP are being used can help us know where to look in docs, examples, etc when this work gets prioritized.

Another thing worth knowing is what PHP version and which version of Swoole or ReactPHP you are using, to make sure our work is actually relevant to what our users need.

[nesl247]

I believe we're using Swoole v4.4.15 at this time with PHP 7.4. And yes, we would expect each new request that swoole processes to be traced independently.

Also, thanks for the fast update.

[SonarSoftware]

Yep, exactly. Just treat each request as independent. I think this might be fairly easy to do by hooking into the onRequest method.. I'm planning to take a look at this myself soon.

[mrtus]

HI @morrisonlevi @labbati Could you provide us with an update on the work being done to make this possible?

And would it maybe be possible to elaborate a little bit more on this:

Even if you manual invoke GlobalTracer::get()->flush() in some after-response hook ..., the Tracer remains in an unpredictable state as it approaches serving next request.

As I'm currently looking into getting our distributed services through queueing connected to each other with tracing (we already have our logs connected through the same way with a trace_id property). I would, just like it is already done with curl, attach the tracing data on the message when published and get that data from the message when consumed by a long running process worker. After this message is consumed, I would of course want to call a flush, but I'm reading here that the tracer would be in an unpredictable state. Can you elaborate what you mean with unpredictable, what are the side-effects of this, and would there be some sort of remenand traces that would tie multiple messages together?

[anderson-xd]

Hi, any updates about this feature?

Thanks!

[labbati]

Hi @anderson-flores! At the moment we were not able to schedule the work for this and we cannot commit to a specific timeline. We will update as soon as possible.

[wregis]

I guess I've got a similar problem with Laravel Horizon running for days on a server. It would be nice to trace jobs individually.

[alexjose]

Swoole and ReactPHP is gaining more and more traction. At the same time, we have Laravel Octane working with Swoole under the hood. It would be very great to speed up the support of the coroutines in Datadog.

[anderson-xd]

Hello, any updates?

A year has passed and more and more swoole is gaining attention from the community, especially after the release of Laravel Octane and other packages with Swoole for Laravel, do we have any estimate when this will be implemented? Or if it won't be

thks.

[bwoebi]

We do not yet have concrete updates on this, except that it is on our roadmap to have at least the possibility of manual tracing of async applications by now (we expect some actual results regarding async surfacing this year) and new APIs are being designed with async in mind (e.g. the usage of a span stack like we currently have is inherently incompatible with async use cases).

Concrete new integrations will be evaluated once support for async instrumenting has landed at all.

[elijahchancey]

We'd really really love to have this!

[chrisbradleydev]

Any news on tracing for Laravel Octane Swoole?

[bwoebi]

I sadly have to repeat what I said in February. Hopefully basic fiber support, including simple async support will land within a couple months. Only then, we will be able to look into laravel octane or swoole.

[chrisbradleydev]

@bwoebi, thanks for the quick reply.

[egonbraun]

Quick question, is this support with RoadRunner instead of Swoole? Or is this not support due to the async nature of Laravel?

[donparapidos]

@bwoebi any news on this, in few weeks it will be 3 years since initial question ? quite some time

[morrisonlevi]

In ddtrace 0.82.0, we added a Roadrunner integration. Bob will be unavailable until the New Year, and I am filling in as best as I can. I don't know what's on the roadmap, or what's blocking more Async/CLI setups, sorry!

[githubeto]

I set up ddtrace 0.82.0 in the Laravel Octane (Roadrunner) application, but still no trace data was output to APM. Does that mean that 0.82.0 is still not supported? And also, when do you plan to release?

[bwoebi]

Hey @githubeto, Minimal tracing data about the requests occurring should appear since 0.82.0 with Laravel Octane running on roadrunner. I'd like to ask whether you did enable CLI? Roadrunner is spawning CLI processes, which needs to be explicitly enabled in ini via datadog.trace.cli_enabled=1 (or DD_TRACE_CLI_ENABLED=1 env var).

[githubeto]

@bwoebi , appreciate your quick reply. I had already set DD_TRACE_CLI_ENABLED=true in my Docker laravel application. Extract a part of the dockerfile.

• dockerfile (laravel application)

  
  RUN curl -LO https://github.com/DataDog/dd-trace-php/releases/latest/download/datadog-setup.php
  RUN php datadog-setup.php --php-bin=all

ENV DD_TRACE_CLI_ENABLED true ENV DD_TRACE_GENERATE_ROOT_SPAN false ENV DD_TRACE_AUTO_FLUSH_ENABLED true ENV DD_TRACE_GENERATE_ROOT_SPAN true ENV DD_TRACE_ANALYTICS_ENABLED true ENV DD_TRACE_LARAVEL_ENABLED true

This is an application running on AWS Fargate, and the task definition also includes the following settings.

- laravel application container

            {
                "name": "DD_ENV",
                "value": "infra"
            },
            {
                "name": "DD_SERVICE",
                "value": "api"
            },
            {
                "name": "DD_VERSION",
                "value": "beta"
            },

- datadog agent container

           {
                "name": "DD_SITE",
                "value": "datadoghq.com"
            },
            {
                "name": "DD_PROCFS_PATH",
                "value": "/proc"
            },
            {
                "name": "DD_APM_NON_LOCAL_TRAFFIC",
                "value": "true"
            },
            {
                "name": "DD_APM_ENABLED",
                "value": "true"
            },
            {
                "name": "ECS_FARGATE",
                "value": "true"
            },
            {
                "name": "DD_LOGS_ENABLED",
                "value": "true"
            },
            {
                "name": "DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL",
                "value": "true"
            },
            {
                "name": "DD_CONTAINER_EXCLUDE",
                "value": "name:datadog-agent"
            },
            {
                "name": "DD_PROCESS_AGENT_ENABLED",
                "value": "true"
            },
            {
                "name": "DD_APM_ENV",
                "value": "infra"
            }

            "secrets": [
                {
                    "name": "DD_API_KEY",
                    "valueFrom": "xxxxxxxxxxxxx"
                }
            ],
```

If I change the application server configuration to nginx + php-fpm instead of laravel octane(roadrunner),
tracing data about the requests occurring appear, so I think the settings I wrote above are fine.

[donparapidos]

4 months passed since I asked and over 3 years since this issue was raised and we still got no answer about the roadmap. Will Datadog support apm on Swoole, ever ? Swoole is now super popular and I cant believe company like Datadog ignores it

[bwoebi]

Hey @donparapidos,

We are sorry for not supporting Swoole yet.

However, your comment just prompted me into looking again into the swoole project. They seem to have committed support for the Zend Fiber API in https://github.com/swoole/swoole-src/commit/a941e94497af6326c72c70e62a031d3efc8c6d3b, which was released in February in v5.0.2.

Given that this support exists now, we are now able to put Swoole support with a higher priority on our roadmap!

Thanks for bringing it to my attention again. :-)

[bwoebi]

I haven't tested it, but I think just judging from that commit, that ddtrace should run on v5.0.2 without crashing (proper instrumentation is still missing, but there's a chance it will run at all).

[s1rc]

With the release of Horizon async queue support in 0.87.0, any chances we will see proper support for Octane soon?

[leonvd6]

Any updates on Swoole support? We migrating from PHP-FPM to OpenSwoole with Laravel Octane and the performance gains are massive but then we lose Datadog support.

[bwoebi]

OpenSwoole does not provide an API for tracking context switches. (Development seems also to be relatively quiet there.)

With Swoole, we're targeting next quarter to have a closer investigation and possibly actual support.

[carlosevieira]

Any updates on this @bwoebi ? I just migrated from php-fpm to Swoole, but I'm just missing datadog :/

[GuillaumeBrook]

Same question, @bwoebi, any progress has been made in Swoole support ?

[carlosevieira]

4 years of no support for one of the popular deployment stacks for PHP applications, maybe this year we will have support for Swolle? @bwoebi @estringana :(

I can't believe a company like Datadog is ignoring this...

[bwoebi]

@carlosevieira Looks like we will have it soon :-)

[PROFeNoM]

Hi everyone :wave:

The 0.99.0 release of the PHP Library added support for Swoole 🎉

Upgrading to the latest version of the tracer will create a trace for each individual request made to Swoole HTTP Servers without you having to perform any manual instrumentation.

Should any issue arise, we encourage you to open a new Github Issue or contact our Support Team for assistance.

We greatly appreciate your patience :pray:

[sneycampos]

Thansk for the release @PROFeNoM! Could you share any instructions to instrumentate the application with Laravel Octane running on Swoole?

Setting php.ini?

[PROFeNoM]

Hi @sneycampos :wave:

Typically, if you already have a Laravel Octane w/ Swoole app, you only have to upgrade the tracer.

If the service is not yet instrumented, then you would have to download and install the tracer and forward it to your datadog-agent.

---

I'll now give an example if this can help you out. Following the official documentation of Laravel Octane, let's say you have set up your application with Laravel Sail. You'll certainly have a docker/ directory or such with your various Dockerfiles. To install the tracer, you would add the following two lines to your Dockerfile:

curl -LO https://github.com/DataDog/dd-trace-php/releases/latest/download/datadog-setup.php
# APM only
php datadog-setup.php --php-bin=all

and in your docker-compose.yml, you'd typically have to add at the very least the following environment variable to your Laravel service:

DD_TRACE_CLI_ENABLED: true
DD_AGENT_HOST: datadog-agent

and a datadog-agent service:

  datadog-agent:
    image: datadog/agent:latest
    container_name: datadog-agent
    environment:
      DD_API_KEY: <YOUR_API_KEY>
      DD_SITE: <YOUR_DD_SITE>
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro

I extracted the above snippets from a sample Laravel Octane app with Swoole of mine, so that's just an example. It will ultimately depend on the specifics of your services, though.

---

I'm happy to answer any specific questions you may have. If you are having trouble instrumenting your app, I greatly encourage you to either contact our Support Teams or create a GitHub Issue.

I hope it helped 🙏

[sneycampos]

I hope it helped 🙏

Thank you @PROFeNoM! Appreciate your help!

I got the same result when tried a few days ago:

made some requests to application, here a few logs

1. I'm using the FROM openswoole/swoole:22.1.1-php8.3-alpine image

2. added the datadog-setup.php

3. my service

4. my datadog agent

In datadog dashboard, i can't see any trace this time, in my tests a few days ago got only the first command (php artisan migrate) that my entrypoint script calls and request logs didn't get anything

It appears in my infrastructure list | Containers

I tried with your octane repository and got same results :/

[PROFeNoM]

Hi @sneycampos

Thanks for coming up with all this information.

I'm using the FROM openswoole/swoole:22.1.1-php8.3-alpine image

I'm afraid this is the root cause of your issue. Your image uses OpenSwoole, while our tracing library provides support for Swoole.

As of today, it is not possible to provide support for OpenSwoole because it does not support Fibers, which are required for proper tracing and interception.

When it comes to Laravel Octane, we are supporting:

• FrankenPHP
• Roadrunner
• Swoole (which does not include OpenSwoole)

Until OpenSwoole makes changes on their side, I'm afraid there is nothing that can be done on our side to provide proper support for OpenSwoole :(

[sneycampos]

@PROFeNoM Thank you! I'll replace the image to use Swoole's and give you a feedback :)

[sneycampos]

@PROFeNoM could you share any dockerfile/docker-compose with agent and octane working? i tried with your repository (https://github.com/PROFeNoM/octane-demo/blob/main/docker-compose.yml) and didn't work too

Tried too with the following image: FROM phpswoole/swoole:5.1.2-php8.2-alpine