[Bug]: Laravel Octane traces not showing in dashboard

[s1rc]

Bug report

We have a k8s deployment where we are turning Laravel Octane on instead of php-fpm. I have added the followingDD_TRACE_CLI_ENABLED: true to the deployment as per https://github.com/DataDog/dd-trace-php/issues/704#issuecomment-2049490652

We are not seeing any traces from testing calls on the server. We do see artisan config:cache and the nginx trace, which is configured in front of Laravel Octane.

Aside from adding DD_TRACE_CLI_ENABLED: true is there anything else we need to do?

Here are the configured DD specific ENV variables:

DD_DOGSTATSD_URL=unix:///var/run/datadog/dsd.socket
DD_DBM_PROPAGATION_MODE=full
DD_INSTRUMENTATION_INSTALL_ID=5078c538-b063-476e-8d3f-21eaed4de329
DD_ENTITY_ID=8be3d3b3-91e5-49f9-8757-7f1773a250a6
DD_TRACE_CLI_ENABLED=1
DD_INSTRUMENTATION_INSTALL_TIME=1713534130
DD_INSTRUMENTATION_TELEMETRY_ENABLED=0
DD_SERVICE=staging-api
DD_LOGS_INJECTION=1
DD_TRACE_CLIENT_IP_ENABLED=1
DD_TRACE_AGENT_URL=unix:///var/run/datadog/apm.socket
DD_ENV=staging
DD_AGENT_HOST=10.0.38.5
DD_TRACE_CLIENT_IP_HEADER=x-forwarded-for
DD_VERSION=develop-9ac3f0c9
DD_QUEUE_TRACE_SERVICE_PROVIDER_ENABLED=false

PHP version

8.2.17

Tracer or profiler version

0.99.1

Installed extensions

[PHP Modules]
bcmath
bz2
Core
ctype
curl
date
ddappsec
ddtrace
dom
exif
fileinfo
filter
gd
hash
iconv
imagick
intl
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
random
readline
Reflection
session
SimpleXML
sockets
sodium
SPL
sqlite3
standard
swoole
tokenizer
xml
xmlreader
xmlwriter
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
ddappsec
ddtrace

[PROFeNoM]

Hello :wave:

Do you know if the Trace agent is receiving the traces at all? You can see that by either checking the datadog-agent status, or by looking at the Trace Agent dashboard on the Datadig UI for your corresponding service.

Thanks!

[s1rc]

@PROFeNoM The agent is definitely working, there are other services still reporting traces within that cluster.

There are other operation traces coming from those service pods but not laravel.request. e.g. If php artisan route:cache is run you see that trace and other operations such as:

laravel.event.handle
command_execution
laravel.provider.load

[PROFeNoM]

Hi @s1rc,

I can reproduce it to keep you updated, but I may have found the root cause. I'm working on a fix/Figuring out the best course of action.

I wanted to know: Was there any particular reason you disabled DD_INSTRUMENTATION_TELEMETRY_ENABLED?

[s1rc]

@PROFeNoM awesome, if there's any way I can help let me know.

Re: DD_INSTRUMENTATION_TELEMETRY_ENABLED it's mostly due to company infosec policies for some environments.

[PROFeNoM]

Right, okay. Thanks for letting me know @s1rc :smiley:

Please try enabling DD_TRACE_SIDECAR_TRACE_SENDER. I started an environment from scratch this morning and reproduced the issue. After trying multiple combinations, enabling this feature fixed the problem. However, I'm still trying to figure out the exact root cause.

Technically, it will send the traces through the sidecar. That's why I was asking about DD_INSTRUMENTATION_TELEMETRY_ENABLED, as telemetry also makes use of the sidecar, and I wanted to make sure there weren't any technical limitations :)

[s1rc]

@PROFeNoM I tried adding DD_TRACE_SIDECAR_TRACE_SENDER and I didn't see any difference. Even enabling DD_INSTRUMENTATION_TELEMETRY_ENABLED in this staging environment it doesn't show any laravel.request traces.

Interesting thing, I notice that now I am seeing traces for artisan octane:start only after the pod is terminated, and that trace has a duration that matches the lifespan of the pod:

[PROFeNoM]

Hi @s1rc :wave:

Thanks for your patience.

An issue was attempted to be fixed with #2645. Could you please try out the following artifact (CircleCI Job)?

---

To use the artifact, if you followed the public doc, then you have to replace the link you were using to download the datadog-setup.php installer script by the one from the artifact and rebuild :+1:

---

Just so you know, I was using the following app to do my tests.

[sneycampos]

Hi @PROFeNoM, did some tests using your repository and got the same results

called a few times (100+) the simple endpoint, simple_view and nothing...

The error endpoint getting the exception

[sneycampos]

Noticed that in APM > Traces i can see some data

But in APM > Service Catalog has nothing

[PROFeNoM]

Hey @sneycampos, in the service catalog, can you please look for web.request as the operation name?

[sneycampos]

Hey @sneycampos, in the service catalog, can you please look for web.request as the operation name?

Ok, now i see :D, there's any way to set web.request as primary operation? laravel.request doesn't see useful for us (customer) i think

Now i'll try to figure out how to work in my application (using alpine images) a few minutes before trying your repository i tried with my app using openswoole/swoole:22.1.1-php8.3-alpine image, installed libgcc but doesn't see any web.request

[s1rc]

@sneycampos I don't think OpenSwoole is supported re: https://github.com/DataDog/dd-trace-php/issues/704#issuecomment-2051428414

Also, from the service page click on "Service Info" top right corner. There is a tab "Primary Operation".

[sneycampos]

Sorry, forgot this message. I will switch to swoole's image and give you a feedback as soon as possible

[sneycampos]

Ok, now it works! a great news this working with laravel octane finally :rocket:

Hope it is release asap to be used in production

Great work @PROFeNoM!

[s1rc]

Thank you @PROFeNoM , so far with that artifact I am seeing traces under the web.request operation! I will keep testing tomorrow.

I do have follow up questions:

• Is it not possible to still have octane requests under the laravel.request operation if we added config for that? e.g. ENV DD_*
• If the operation name has to be different, will that affect metrics sources? e.g. We are using metrics such as trace.laravel.request.hits.by_http_status on monitors and SLOs, etc. Will we have to change all of those if we are using Octane?

[bwoebi]

The 1.0 is now out, containing the support for this :-)

[tkxel-owais]

Adding "support" via DD_TRACE_CLI_ENABLED is not exactly "adding" support.