Proxying metrics with auth via headers

[ldaneliukas]

We're trying to scrape the control plane metrics of Kubernetes, e.g. the kube-apiserver that exposes metrics on localhost:6443/metrics but that requires authorization via the bearer-token header when using RBAC. (Metrics For The Kubernetes Control Plane)

This could be done by a dedicated job in Prometheus, however, when exporter_exporter is already used to proxy other exporters on the same machine, we'd prefer to do the same here.

That's only a single example derived from an actual use case that we've encountered. Implementing this would require a simple modification to the reverse proxy so that headers from the module config (if any) are added when proxying, would this be acceptable?

[tcolgate]

To clarify, you want to be be able to specify some full headers (name and value), in the config file and have those added to incoming requests? Or you want to be able to white list headers that would be passed through?

[ldaneliukas]

I was thinking about allowing key/value pairs in the module config that would be passed as headers when proxying. If we look at the use case that we've encountered, we can be scraping any number of kube-apiserver and other endpoints on various machines that can require different auth tokens in the headers. Creating separate scrape jobs that would pass different tokens via headers wouldn't be ideal, hence, the module config having them would work better in this case.

[tcolgate]

Would bearer token be enough? My thinking is, having a bearer token in a file, referred to in the config, similarly perhaps basic auth support. Might be preferable to baking secrets into the config file for some users. I think having extra headers is fine, and would handle your use case. But if we introduce it, I might be inclined to introduce additional support specifically for reading in bearer tokens and basic-auth passwords from a file. I may not have explained that well. Go ahead with a PR for the header support, I can look at adding specific Authorization support later.

[ldaneliukas]

I've added custom header support.

What you mean is having the ability to specify something like:

modules:
  node:
    method: http
    http:
       port: 9100
       auth:
          bearer: "/tmp/file/with/token"
          basic: "/tmp/file/with/creds"

Am I right?

That would make sense to me for handling auth in cases where you wouldn't want to add tokens to the actual config file for expexp