Wrong percentile values for metrics shown in Datadog

imiric commented 1 year ago

Brief summary

As reported several times on the forum (1, 2, 3), the percentile values users can visualize in Datadog are often wrong when compared to the end-of-test summary shown by k6.

As explained here, this is likely caused by the additional aggregation done by the DogStatsD agent, which by default only generates the 95percentile metric, which is then aggregated again by the query used in the Datadog graph.

Besides resulting in wrong values, users are unable to use raw metric data to generate any percentile value they need.

k6 version

0.41.0

OS

any

Steps to reproduce the problem

Follow the steps to setup the Datadog Agent, and run a test as explained here.
Go to the Datadog web UI and visualize the k6.http_req_duration.max and k6.http_req_duration.95percentile metrics, and compare them to the end-of-test summary shown by k6. Notice that they don't match.

Expected behaviour

The percentiles shown in Datadog should match the end-of-test summary shown by k6.
The user should be able to generate any percentile over raw metric data sent to Datadog. Ideally, the Datadog Agent (DogStatsD) shouldn't do any aggregation at all.

Actual behaviour

The percentiles are different, and the Datadog 95percentile is confusing.
The user can only use a limited number of metrics, and most pre-aggregated ones show wrong values.

Suggested solution

After going through the Datadog documentation, it seems it's possible to send raw data that won't be aggregated by the Datadog Agent using the distribution metric type (1, 2). This would not only avoid the aggregation, but allow users to generate any percentile value they need over the raw data.

The drawback would possibly be overloading the ingest pipeline (either of the Datadog Agent or of Datadog itself, hitting some API limits, etc.), so this needs to be thoroughly tested.

Currently we send Count, Gauge and other metric types that will be aggregated, but the datadog-go/statsd client we use also supports the Distribution metric.

In addition to evaluating whether this change works for some of our metric types, we should also ensure that we don't break support for any other StatsD backends.

imiric commented 1 year ago

I think sending our Trend metric as a Distribution metric would also fix querying percentiles in New Relic. See the New Relic data types documentation, specifically for the distribution type. Notice that it supports the percentile and histogram functions, which are not supported by the other types.

Currently this is not supported, and we removed the documentation about the previously wrong query in https://github.com/grafana/k6-docs/pull/895.

vieiraes commented 1 year ago

waiting for a update about it... all % metrics are different in Datadog...

imiric commented 1 year ago

Sorry about that @vieiraes. :disappointed: You can see some updates in #2982.

What we decided to do is to split up the DataDog and New Relic outputs into separate k6 extensions, since it's clear that they can't be well supported by the generic statsd output.

This work is high priority, as we know it's a major impact for k6 users of these backends, so it's likely we'll start working on this in the next few weeks, but we can't promise any ETAs yet.

LeonAdato commented 1 week ago

Per @olegbespalov and @javaducky, this issue should probably be part of the StatsD project. Feel free to transfer it here:

https://github.com/LeonAdato/xk6-output-statsd

olegbespalov commented 1 week ago

Closing in favor of https://github.com/LeonAdato/xk6-output-statsd/issues/24

grafana / k6