DataDog / saluki

An experimental toolkit for building telemetry data planes in Rust.
Apache License 2.0
12 stars 3 forks source link

Add full histogram support for DogStatD. #270

Open tobz opened 3 days ago

tobz commented 3 days ago

Context

Currently, our DogStatsD parser, and indeed Saluki's event model, only allow for the possibility of storing distributions: a histogram-like data structure (in the mathematical sense) that can answer arbitrary quantile queries against a set of samples. This happens for all histogram-like input types: timers, histograms, and distributions. We did this because it was a reasonable approximation of the average behavior of these metric types: be able to answer questions about the distribution of values observed.

However, in the Datadog Agent, histograms are treated separately (by default) where pre-determined aggregations/quantiles are calcuated and emitted as individual metrics, rather than sending the entire distribution to the Datadog backend for generic querying.

We need to be able to match this behavior such that we can emit histograms in the same way for compatibility with the Datadog Agent.

tobz commented 3 days ago

In my mind, the most reasonable design is one that involves adding a new variant to MetricValues, called Histogram, that holds a DDSketch. I believe this is the most reasonable design for a few reasons: