search

DataDog / lading

A suite of data generation and load testing tools
MIT License
76 stars 11 forks source link

Enable tagging of target metrics from prometheus and expvar endpoints by sub-agent name #941

Closed cmetz100 closed 2 weeks ago

cmetz100 commented 4 weeks ago

What does this PR do?

This PR adds another configuration option to prometheus and expvar target metrics enabling a user to tag all the metrics coming from that configuration with optional additional labels

target_metrics:
  - prometheus: #process agent telemetry 
      uri: "http://127.0.0.1:6062/telemetry"
      tags:
        sub_agent: "process"
  - prometheus: #core agent telemetry
      uri: "http://127.0.0.1:5000/telemetry"
      tags:
        sub_agent: "core"

Motivation

Currently in lading we could configure several prometheus endpoints to listen on to collect target metrics. In the example case of the agent being the target we could configure separate prometheus target metrics to get the internal telemetry of different sub agents. This issue is that when we do this there are some duplicate metrics across the sub agents. For example both the core agent and process agent expose workloadmeta telemetry. We want a way to differentiate not just the duplicate metrics but all of them, so that we can investigate the performance of each sub-agent on its own. This extends to any target. It would be useful to have additional labels configured to differentiate between target metric sources. The use of this applies beyond the agent target. The ability to tag metrics from specific target metrics endpoints would allow more flexibility and observability for any target.

Related issues

SMPTNG-422

Additional Notes

Anything else we should know when reviewing?

Open Questions: -Should I include two labels in the example in the changelog? We only know of use cases at the moment using one additional tag but it is not really clear from the code that you can configure multiple. I also wanted to emphasize that the user can create the tag name by setting whatever key value they want, in addition to setting the tag value.

cmetz100 commented 4 weeks ago

I like the overall approach, lets change the sub_agent terminology to be something more generic. lading is intended to be "target agnostic", so the target could be the Datadog Agent, but it could also be Vector or any other arbitrary program.

It also appears that there may be missing files from this branch, I would guess there are changes incoming in target_metrics/expvar.rs as well as lading.rs

I was thinking about expanding the functionality once i got this working to enable the addition of a arbitrary amount of key value labels so that we could configure appropriate label names for whatever target. And yeah working on the expvar changes right now.