[Feature] Allow Prometheus.scrape to authenticate using `meta` label information from discovery components

[proffalken]

Related to grafana/agent#5233, I'm working on a way to get Bearer Token or other auth methods to be passed dynamically to my Grafana Agent running flow.

I've written a small microservice in Django that I'm running with discovery.http on my local network, and I keep all the services that aren't in K8s listed in that service:

The output of the service is a JSON array of hosts, metricsPaths, ports, etc. that can be consumed by discovery.http but as soon as any of those endpoints need an auth token, I have to add a new prometheus.scrape with the token details, and at that point the scraper stops being generic.

The output is as follows at present:

[
   {
      "labels" : {
         "__metrics_path__" : "/metrics",
         "__scheme__" : "http",
         "__scrape_interval__" : "60s",
         "__scrape_timeout__" : "10s",
         "service" : "PiHole"
      },
      "targets" : [
         "192.168.12.1:9100"
      ]
   },
   {
      "labels" : {
         "__metrics_path__" : "/api/prometheus",
         "__scheme__" : "https",
         "__scrape_interval__" : "60s",
         "__scrape_timeout__" : "10s",
         "service" : "HomeAssistant"
      },
      "targets" : [
         "192.168.12.10:443"
      ]
   }
]

And this works well for discovery.http and a generic prometheus.scrape apart from the auth tokens.

Ideally, I'd be able to do something like this:

[
   {
      "labels" : {
         "__metrics_path__" : "/metrics",
         "__scheme__" : "http",
         "__scrape_interval__" : "60s",
         "__scrape_timeout__" : "10s",
         "service" : "PiHole",
         "__meta__bearer_token": <MY TOKEN>
      },
      "targets" : [
         "192.168.12.1:9100"
      ]
   },
   {
      "labels" : {
         "__metrics_path__" : "/api/prometheus",
         "__scheme__" : "https",
         "__scrape_interval__" : "60s",
         "__scrape_timeout__" : "10s",
         "service" : "HomeAssistant"
      },
      "targets" : [
         "192.168.12.10:443"
      ]
   }
]

and for those targets that support it, prometheus.scrape would pick up the token from __meta__bearer_token and use it to authenticate against the appropriate target host and metricsPath

Is there any chance of this functionality being added to the prometheus.scrape component? I realise it moves it away slightly from "standard" Prom labels, however Prom does allow for __meta__ labels as part of the relable_config so I'm hoping we can take advantage of that!

_Originally posted by @proffalken in https://github.com/grafana/agent/pull/5233#discussion_r1331275071_

[proffalken]

Adding some more context following a discussion with @mellieA on Slack.

Question was, is https://stackoverflow.com/questions/67847139/how-to-add-bearer-token-for-prometheus-job similar?

Answer:

This is kind of the same thing, just a different way to deliver it.

The example in the answer to that SO question has all the bits you need to make a request:

- job_name: 'test'
    metrics_path: "/metrics"
    scheme: "http"
    bearer_token_file: /var/run/secrets/    OR   bearer_token: token_here
    static_configs:
- targets: ['host.com']

At the moment, I can set the metrics path, the scheme, and the targets all via the _meta tags, but I can't set bearer token. I'll update the issue in a bit, but what I'm essentially proposing is that __meta_bearer_token from discovery.http carries the value that prometheus.scrape needs to authenticate, removing the need for Flow to reach out to the authentication service that the target application uses and retrieve a token from there as the question suggests.

I guess you could take the approach in that question, but that would still mean you'd need to authenticate somewhere and you'd still need to have credentials stored that discovery.http could use to authenticate via BasicAuth or similar to retrieve the token, and not all API's that use Bearer Tokens will necessarily have an endpoint that automatically returns that token

[github-actions[bot]]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

[rfratto]

I'm a little concerned with the idea of meta labels being used for sensitive information; labels are inherently currently non-sensitive, and there's a risk of leaking secrets if sensitive information is stored in labels.

I think we need to think about this idea more, and identify if there's more secure ways to allow more flexible ways to authenticate against services for telemetry collection.

[proffalken]

I'm a little concerned with the idea of meta labels being used for sensitive information; labels are inherently currently non-sensitive, and there's a risk of leaking secrets if sensitive information is stored in labels.

That's a great point, and a very valid one.

I think we need to think about this idea more, and identify if there's more secure ways to allow more flexible ways to authenticate against services for telemetry collection.

Yup, more than happy to chat about that!

[proffalken]

@rfratto - I've been playing around with the new metrics endpoint integration and I'm now wondering if the issue here is something that we should be promoting as a use-case for Agent at all?

Chatting with @matthewnolf today I've realised that (as far as Grafana Cloud is concerned anyway), creating a new Metrics Endpoint with these details via Terraform or similar would probably be a lot more straight forward than trying to ensure that secrets are kept secure whilst also adhering to the Prom labels format as you point out above.

Worth further discussion I think?