[ruler] Support external labels

[bilbof]

Is your feature request related to a problem? Please describe.

It's not possible to set extra labels on series at the ruler, like it is with Prometheus with external_labels. So you can't set e.g. 'cluster' or 'environment' etc.

Describe the solution you'd like

Something like what Prometheus provides:

global:
  # The labels to add to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    [ [<labelname>](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#labelname): [<labelvalue>](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#labelvalue) ... ]

[pracucci]

👋

So you can't set e.g. 'cluster' or 'environment' etc.

Could you share more about your use case, please? Why would you want to set, for example, "cluster" on the ruler series? Mimir typically is deployed as a centralised metrics database, collecting metrics from multiple clusters, so the "cluster" label is expected to be set in ingested metrics (received from Prometheus or the Agent) and then the ruler just use it.

My previous question follows the "cluster" example you made, but generally speaking we haven't seen the need of having external labels support in the Mimir ruler so far, so I would like to better understand your use case. Thanks!

[tgrondier]

Hi! @bilbof has not answered but I will share our use case:

We have a single dashboard setup as a receiver of all our alertmanager in different environments (testing, staging, production, etc). We rely on external labels sent by prometheus to alertmanager to separate and categorize alerts automatically in the dashboard. Without that, all alerts are grouped together as if they originated from the same environment

EDIT: We 'fixed' it by using mimirtool rules prepare

[jleloup]

A similar use cas I have that could benefit from having external labels set bu Mimir Ruler:

I am using Sloth to produce PrometheusRules resources which are then loaded into Mimir Ruler via Grafana Agent mimir.rules.kubernetes. During this process I make sure to load these rules into all my Mimir tenants (which are representations of my environments producing metrics)

The problem then is that I am relying on a label environment in my alerts to be used later on for alert routing: think different Slack channels per environment for example. Usually Prometheus external labels are enough to do the trick though in the context of these SLO rules:

• there is no possibility to add such external label within Mimir Ruler (neither in Grafana Agent during the provisioning of those rules into Mimir Ruler)
• Since I am not handling the rules directly -this is done by SLoth): I don't have the possibility to either add this label explictely or tweak the PromQL queries to keep it from the raw metrics

[QuentinBisson]

We have the same problem and the issue is that with prometheus we crafted a lot of alerts using aggregations (like sum by) and used external labels to add those labels in alertmanager but with Mimir, we would need to add those labels to all by clauses of all alerts which makes them then really hard to read when you have more than just 1 external label