[Feature] Add APM for Serverless PHP

[eddmann]

Over the past couple of weeks we have been setting up DataDog APM within our stack. We have found great value in enabling this on our conventional server-based PHP instances, using the DataDog Agent to forward on the tracing to DataDog.

However, over the past year we have been adopting Serverless (via AWS Lambda) - successfully running PHP applications brought in large part by the BrefPHP project. We noticed that Serverless support is present for languages such as Python/Ruby but sadly not for PHP. We really liked the idea of getting the same level of insight into our Serverless environment as we did the conventional server-based approach so we set aside sometime to explore if this would be possible.

Proof-of-concept

We were able to create a PoC based on the existing PHP-APM extension along with a couple of small modifications. We have made this available at https://github.com/mybuilder/datadog-php-tracer-lambda-layers.

At a technical level you can see that based on how the Tracer has been designed we only need to supply a different Transport which writes these traces out to stdout (CloudWatch Logs) - like what is done in other languages such as Python. We are already using this to supply custom-metrics from our Serverless application using the JSON-based format which is processed by the DataDog Forwarder.

As it stands this works surprisingly well, with only a couple of caveats:

• We cater for dropping traces within the Transport as opposed to always sending onto the DataDog Forwarder so as to not incur extra cost - we have not invested the time to explore how Python etc. does this, or if they simply send all traces along to DataDog for them to be discarded and metrics to be counted.
• We had to modify the format in which identifiers were sent (strings instead of integer) as the DataDog endpoint did not like this.
• We found we needed to chunk up the traces being sent out on the stdout for CloudWatch as we seemed to hit a max-line limit.
• We are having issues with distributed traces crossing the Serverless/Server-based divide, Serverless-to-Serverless traces are working as expected but not between. We have not been able to figure out what this issue is due to.

Conclusion

At a high-level we are wondering if this is something you would look to support in the future?

We understand that PHP is not a supported runtime for AWS so providing a layer etc. maybe a little non-trivial, but providing a means to supply a stdout based trace output like we have shown would be great! In regards to this there already is a Stream transport and Json encoder present in the codebase, so we are not sure if this was an intention at some point already?

[lbassin]

Hi guys, is there any update on this issue? Would you be open to a MR to implement the proof of concept that @eddmann provided?

[bwoebi]

Yes, we would be open to such a MR. However, such an implementation would have to be at the extension layer by now.

That implementation would need to hook into https://github.com/DataDog/dd-trace-php/blob/38c9ca6765cb6430e8bcee802aa539f7994ddf4b/ext/php8/auto_flush.c#L9.

Note that we are also open to providing an API, such that one can provide a callback from PHP code to be invoked upon flushing here.

[deleugpn]

Just wanted to share this here in case someone else in the future finds it useful:

https://blog.deleu.dev/datadog-aws-lambda-php/

TL;DR it's possible to use Datadog on Lambda without changing anything on the PHP Extension

[dzavalkin-scayle]

We managed to get it working in a slightly different way. We run Datadog agent as a Fargate task and install only dd-tracer-php PHP extension inside lambda and configured it to connect to external Datadog agent. This way we avoid any possible performance hits caused by running Datadog agent inside lambda. This setup is not in production yet, we are still testing but no problems were noticed so far - dd-tracer-php sends all spans to external Datadog agent and their appear in Datadog UI as expected.

[deleugpn]

hey @dzavalkin-scayle mind sharing how do you configure your Lambda to send logs to the Datadog Agent on Fargate?

[zchermit]

@dzavalkin-scayle Also interesting how to configure this solution.

[ttrushin]

@dzavalkin-scayle Same here. Would love to know how you guys set this up and whether or not it ever made it to production.

[danshou]

It would be great to have this feature to monitor PHP in Lambda.

[Anixed]

With my team we need this feature for serverless PHP.

@dzavalkin-scayle How did with this configuration in production?

[dzavalkin-scayle]

Setup I described above is for APM Datadog integration. Datadog is running as an ECS/Fargate task/service and there is an NLB pointed to it to port 8126. On Lambda side we set the following env variables:

DD_AUTOFINISH_SPANS: true
DD_ENV: envname
DD_SERVICE: servicename
DD_SERVICE_MAPPING: pdo:servicename-envname-rds,phpredis:servicename-envname-ec,redis:servicename-envname-ec,guzzle:servicename-envname-guzzle,curl:servicename-envname-curl
DD_TAGS: service:servicename
DD_TRACE_AGENT_URL: http://servicename-datadog-nlb-int-1234567890123456.elb.eu-west-1.amazonaws.com:8126
DD_TRACE_AUTO_FLUSH_ENABLED: true
DD_TRACE_CLI_ENABLED: true
DD_TRACE_CURL_ENABLED: true
DD_TRACE_ELOQUENT_ENABLED: true
DD_TRACE_ENABLED: true
DD_TRACE_GENERATE_ROOT_SPAN: true
DD_TRACE_GUZZLE_ENABLED: true
DD_TRACE_LARAVEL_ENABLED: true
DD_TRACE_PDO_ENABLED: true

To forward Lambda logs (and also logs from ECS/Fargate tasks) we use Datadog Forwarder (i.e. CloudWatch Logs -> DD forwarding): https://docs.datadoghq.com/logs/guide/forwarder/

[koullislp]

Would be great to have this supported by Datadog!

[astuyve]

Hey folks! Quick poll on runtimes in Lambda here: 🎉 - Using Bref FPM 🌮 - Using Bref Functions 🚀 - Other (please comment/specify)

Thanks!

[ttrushin]

We are using Laravel Vapor.