prometheus-community / prom-label-proxy

A proxy that enforces a given label in a given PromQL query.
Apache License 2.0
248 stars 95 forks source link

Allow different label names injected by an ExtractLabeler #190

Open mutefiRe opened 4 months ago

mutefiRe commented 4 months ago

In a larger system, maintaining consistent tenant identifiers across all metrics isn't always practical. For example, some metrics might use tenant_id while others rely on page_id, depending on the reporting service. Therefore, the proxy needs to decide which label to enforce for each metric.

We currently have a working prototype that modifies vendor files. Instead of using the label parameter in the NewRoutes function, we inject the label name in the same way as the label values. As a result, the ExtractLabelerhandles determining which label name to enforce.

Since we don't require the full functionality of the proxy, such as rules, silences, and alerts, which might be challenging to support, I wanted to see if you're interested in and supportive of this idea. If so, we're open to investing more effort in refining it for a potential upstream pull request for a possible direction. Anyways, we can provide a fork once it's ready.

jotak commented 2 weeks ago

Hi @mutefiRe I'm also interested in exactly this (if I read your suggestion correctly) : being able to override the label to use for injection. I think for security reasons, the list of labels allowed for override should be restricted by config (and perhaps enforced to be tied to some specific metrics only)

.. I was working on a PoC that I haven't had the opportunity to test yet ..

Does that sound aligned with you use case?

mutefiRe commented 2 weeks ago

Hi @mutefiRe I'm also interested in exactly this (if I read your suggestion correctly) : being able to override the label to use for injection. I think for security reasons, the list of labels allowed for override should be restricted by config (and perhaps enforced to be tied to some specific metrics only)

.. I was working on a PoC that I haven't had the opportunity to test yet ..

Does that sound aligned with you use case?

Hey @jotak,

We've switched to allowing multiple key-value(s) pairs using a map. The proxy ensures that each existing matcher is duplicated for every different label key. Additionally, we have a configuration for a custom Enforcer that matches regex patterns with the incoming metric name to avoid injecting every possible combination into each upstream request. However, this isn't feasible for endpoints like /api/v1/label/__name__/values, where the metric names are unknown in advance. As a result, every combination must be injected, potentially leading to quite large requests depending on the number of IDs you want to inject. In the previous version, if nothing matched our rules, we also used to fall back to the default rule or inject an artificial label-value combination that surely doesn't exist. This ensured the API remained 100% compatible with the upstream API instead of returning errors.

You can check out our fork as well, but note that we simplified the code for our use case, removing all regex enforcement, as well as alerts and silences.