beam-telemetry / telemetry_metrics_statsd

Telemetry.Metrics reporter for StatsD-compatible metric servers
https://hexdocs.pm/telemetry_metrics_statsd
MIT License
75 stars 44 forks source link

Datadog formatter - Convert tag value of list to multiple tag values with same key #90

Open davidnsimplifi opened 11 months ago

davidnsimplifi commented 11 months ago

To give some context, datadog supports receiving multiple tag values with the same key, so you could have a dogstatsd datagram such as:

application_name.metric_name.event.amount:1|c|#type:failed,reason:reason1,reason:reason2

There's some work that I'm involved with on migrating an elixir application from https://hexdocs.pm/statix/Statix.html to use the telemetry module and this library for recording telemetry, and sending them to datadog. statix supported a list of tags, which could be ["tag1:value", "tag1:secondvalue", "tag2"].

With the telemetry module and telemetry_metrics_statsd, we can send a telemetry metric with metadata such as:

%{type: failed, reason: [reason1, reason2]}

The reason list is being converted to a string by the datadog formatter, and then the datagram looks like:

application_name.metric_name.event.amount:1|c|#type:failed,reason:reason1reason2

The main downside of that is that in datadog dashboards, the reason field is one string, so it's not as clean looking. When you have multiple tag values, you can do queries on reason:reason1, compared to now doing a string search of reason:reason1. As well, when displaying a table of the tag, it will show up as reason1, reason2 automatically if there's multiple tag values.

The changes in this pull request detect a list and then take each item in the list as a new tag value so the datagram looks like the first example of

application_name.metric_name.event.amount:1|c|#type:failed,reason:reason1,reason:reason2

If these changes shouldn't be the default, it could also be put behind a config parameter when setting up TelemetryMetricsStatsd