DataDog / dd-trace-php

Datadog PHP Clients
https://docs.datadoghq.com/tracing/setup/php
Other
481 stars 149 forks source link

[Bug]: WordPress Integration is incomplete #2713

Open Veraxus opened 2 weeks ago

Veraxus commented 2 weeks ago

Bug report

The WordPress integration has a few major problems...

  1. Missing Hook Types: It evaluates only actions. WordPress has two kinds of hooks: actions and filters. And technically, they are all filters - actions are an alias for filters with no return value.

  2. Missing Hooks: It does not trace all hooks that are triggered. Instead, it only evaluates a very, very, very tiny list. As a result, there is no way to identify hooks that are misbehaving, because you don't know what you don't know. Because both custom and dynamic hooks exist, the possible list of hooks is literally infinite. DDTrace must be able to trace any and every hook whenever it is triggered.

Expected Behavior: ALL hooks (actions and filters) are automatically traced.

PHP version

8.2.19

Tracer or profiler version

1.1.0

Installed extensions

No response

Output of phpinfo()

No response

Upgrading from

No response

bwoebi commented 2 weeks ago

Hey @Veraxus,

we have a DD_TRACE_WORDPRESS_ADDITIONAL_ACTIONS env / datadog.trace.wordpress_additional_options ini config option, which allows you to trace additional actions. It is not possible to have a wildcard there though, currently.

It's been a year since we implemented that, but as far as I recall, there were concerns about too many spans being generated. There exist limits (set by the backend) on how many spans we can generate. Also, collecting everything may increase the cost for our users, so it's probably not a good default.

What we wanted to / are going to do is collecting metrics about how often each hook is called.

To catch misbehaving hooks (in terms of performance - which is what I suppose you mean), I can recommend using the profiler and have a quick look there whether anything which shouldn't take much performance is.


If you want to try it manually though, you can edit /opt/datadog/dd-library/<version>/src/DDTrace/Integrations/WordPress/WordPressIntegrationLoader.php and remove the two checks for isset($interestingActions[...]) in that file. probably you'll have to increase the datadog.trace.spans_limit INI setting too.

Is that the experience you wish to have?

PROFeNoM commented 2 weeks ago

Hi @Veraxus :wave:

If I may add some data on this decision, we observed that specifying some core, "interesting" actions as part of the WordPress loading procedure allowed to provide a first layer of observability of the application flow while greatly reducing the number of spans, most of which were noise to the understandability of the flame graph.

Quantitatively speaking, on a custom application with ~20 plugins, the number of spans being sent was reduced by 98.13% while only reducing the represented execution time by 5.96%, from 83.9% to 78.9%.

Filter Hooks are numerous, short (μs) operations that transform data; they are numerous enough to smash the span limit, even when setting it to 10k+.

Besides, I believe @bwoebi provided you with some reasonable options. With that said, we are absolutely open to your feedback and hearing about any use case of yours.

Thanks :-)

sean1615 commented 2 weeks ago

Does this happen in other languages as well? Where Datadog chooses whether to add functuionality or not because it "could be noisy", also isn't there a setting in Datdog where we can specifically limit the number of spans that get pushed through by a certain percentage to lower cost on the customer end? Also, couldn't you feature flag filters as an additional and set a warning that this may increase cost to the customer? I feel that this could be hindering insight into potential bottlenecks that we can see.

morrisonlevi commented 2 weeks ago

:wave: Profiling engineer here. It sounds to me like you want profiling. Profiling will find slow functions regardless of whether they are plugins or not.

Note that profiling does have some WordPress specific knowledge about how plugins are commonly laid out, so the graphs can be colored, filtered, etc by library. Feel free to message me on Datadog's public slack with specific questions, or if you'd like me to give you a tour of profiling on WordPress data.