Support Datadog Distribution metric type ?

segmentio / stats

Go package for abstracting stats collection

https://godoc.org/github.com/segmentio/stats

MIT License

208 stars 32 forks source link

Support Datadog Distribution metric type ? #120

Closed sebcante closed 3 years ago

sebcante commented 4 years ago

Hi @achille-roussel,

Love this stats lib. Would you be interested if i attempt to make a PR to support Datadog Distribution metric types ? This allow global percentiles for any tag combination in Datadog.

From this lib point of view it would be exactly as Histogram but the Datadog client would send with a d instead of h for histogram.

I am thinking adding a new metric type called distribution.

Let me know thanks! -seb

achille-roussel commented 4 years ago

Hello @sebcante!

We always welcome new contribution, thanks for starting the conversation!

I think there are two ways to go about this:

since it's a datadog-specific and is similar to histograms, we could make it a configuration option of datadog.Client, the upside is it doesn't require API changes in programs, but it prevents using both histograms and distributions on the same datadog client.
we add a new API to emit distribution metrics, we'll have to think about how to represent those in backends other than datadog.

What do you think?

sebcante commented 4 years ago

Thanks @achille-roussel very good pointers 👍 !

It is tempting to add a config in the datadog.Client to specify to use datadog distribution for metric type Histogram as it is an implementation details of datadog indeed. That said from what i could understand Distribution comes with limitations:

we can use only up to 10 tags for a specific metric for percentile aggregation
we have to preregister the aggregation in DD dashboard

I am also enquiring at the moment with Datadog if it has a cost implication, that might push the consumer of this lib to only hand pick a couple histogram as distribution. I ll comeback to you shortly when i have the full picture once i talk to Datadog. Stay tuned

bhavanki commented 3 years ago

Resolved, at least as a start, via https://github.com/segmentio/stats/pull/132 . Instead of using a simple flag to send all histograms either as histograms or distributions, this change lets you pick prefixes for histogram metric names to be sent as distributions. This way you can have both types.

There are probably other ways to do this, including supporting distributions fully across the stats library, so other / better ideas are definitely welcome.