Multi-tenancy with ruler endpoint

[valleedelisle]

Describe the bug

When hitting the /prometheus/api/v1/rules endpoint with a X-Scope-OrgID using multiple-tenants, mimir returns a no valid org id found

When retrying the exact same call with only one tenant, it works.

To Reproduce

1. Start Mimir, version 2.9.0 (revision: 761114d)
2. Ingest data with multiple tenants
3. Hit the rules endpoint with a pipe-separated X-Scope-OrgID
4. Current result

   
   $ curl --location 'https://mimir.example.com/prometheus/api/v1/rules' --header 'X-Scope-OrgID: mimir-poc-tenant1|mimir-poc-tenant2|mimir-poc-tenant3' --header 'Authorization: Basic dGVzdDpkR1Z6ZERwMFpYTjBDZz09Cg==' --header 'Cookie: xxx' --insecure  

{"status":"error","data":null,"errorType":"server_error","error":"no valid org id found"}

#### Expected behavior

$ curl --location 'https://mimir.example.com/prometheus/api/v1/rules' --header 'X-Scope-OrgID: mimir-poc-tenant1|mimir-poc-tenant2|mimir-poc-tenant3' --header 'Authorization: Basic dGVzdDpkR1Z6ZERwMFpYTjBDZz09Cg==' --header 'Cookie: xxx' --insecure

{"status":"success",......}

#### Environment

- Infrastructure: k8
- Deployment tool: helm

#### Additional Context
I'm no expert here but it looks like the `ruler/api` class is expecting multiple times to get a single tenantID [[1]](https://github.com/grafana/mimir/blob/main/pkg/ruler/api.go#L158) [[2]](https://github.com/grafana/mimir/blob/main/pkg/ruler/api.go#L266) [[3]](https://github.com/grafana/mimir/blob/main/pkg/ruler/api.go#L369), same for the `ruler/ruler` class [[1]](https://github.com/grafana/mimir/blob/main/pkg/ruler/ruler.go#L923) [[2]](https://github.com/grafana/mimir/blob/main/pkg/ruler/ruler.go#L975) [[3]](https://github.com/grafana/mimir/blob/main/pkg/ruler/ruler.go#L1177)

A similar [issue](https://github.com/grafana/loki/issues/7696) was addressed in loki with this commit: https://github.com/grafana/loki/commit/ad2260aec2d4d587c50a3dfb68d2c89ed2bf3157

#### Logs from the `ruler` service in mimir

ts=2023-09-13T15:40:09.699138866Z caller=log.go:194 level=debug msg="Stream connection from=10.130.0.142:44154" ts=2023-09-13T15:40:10.170818328Z caller=log.go:194 level=debug msg="Initiating push/pull sync with: gem-oss-poc-alertmanager-0-fe4ccbeb 10.130.0.142:7946" ts=2023-09-13T15:40:12.804139091Z caller=api.go:161 level=error user=mimir-poc-tenant1|mimir-poc-tenant2|mimir-poc-tenant3 msg="error extracting org id from context" err="multiple org IDs present" ts=2023-09-13T15:40:12.804232601Z caller=logging.go:111 level=warn traceID=4db0a2321851ad9b msg="GET /prometheus/config/v1/rules?subtype=mimir (500) 160.756µs Response: \"{\\"status\\":\\"error\\",\\"data\\":null,\\"errorType\\":\\"server_error\\",\\"error\\":\\"no org id\\"}\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: Grafana/9.5.3; X-Grafana-Referer: ; X-Scope-Orgid: mimir-poc-tenant1|mimir-poc-tenant2|mimir-poc-tenant3; " ts=2023-09-13T15:40:12.804237564Z caller=logging.go:111 level=warn traceID=73b18b78b85ca54c msg="GET /prometheus/api/v1/rules (500) 154.481µs Response: \"{\\"status\\":\\"error\\",\\"data\\":null,\\"errorType\\":\\"server_error\\",\\"error\\":\\"no valid org id found\\"}\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: Grafana/9.5.3; X-Grafana-Referer: ; X-Scope-Orgid: mimir-poc-tenant1|mimir-poc-tenant2|mimir-poc-tenant3; " ts=2023-09-13T15:40:17.155559899Z caller=logging.go:101 level=debug traceID=1062749fdf060b0c msg="GET /ready (200) 57.249µs" ts=2023-09-13T15:40:21.448455424Z caller=api.go:161 level=error user=mimir-poc-tenant1|mimir-poc-tenant2|mimir-poc-tenant3 msg="error extracting org id from context" err="multiple org IDs present" ts=2023-09-13T15:40:21.448505355Z caller=logging.go:111 level=warn traceID=76fc04e2749f7134 msg="GET /prometheus/config/v1/rules?subtype=mimir (500) 87.245µs Response: \"{\\"status\\":\\"error\\",\\"data\\":null,\\"errorType\\":\\"server_error\\",\\"error\\":\\"no org id\\"}\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: Grafana/9.5.3; X-Grafana-Referer: ; X-Scope-Orgid: mimir-poc-tenant1|mimir-poc-tenant2|mimir-poc-tenant3; " ts=2023-09-13T15:40:21.448520875Z caller=logging.go:111 level=warn traceID=5e387507e574e6d7 msg="GET /prometheus/api/v1/rules (500) 129.502µs Response: \"{\\"status\\":\\"error\\",\\"data\\":null,\\"errorType\\":\\"server_error\\",\\"error\\":\\"no valid org id found\\"}\" ws: false; Accept-Encoding: gzip; Connection: close; User-Agent: Grafana/9.5.3; X-Grafana-Referer: ; X-Scope-Orgid: mimir-poc-tenant1|mimir-poc-tenant2|mimir-poc-tenant3; "

[56quarters]

Can you describe why you'd want to get rules for multiple tenants at the same time? Mimir supports querying multiple tenants at the same time but not writes or manipulation of rules.

[valleedelisle]

Can you describe why you'd want to get rules for multiple tenants at the same time? Mimir supports querying multiple tenants at the same time but not writes or manipulation of rules.

So if we have datasources with multiple tenants, we can't create rules on them. We'd need to either create a datasource for each tenant to alert on them or we should use federated rule groups.

It looks like federated rule groups seems to be achieving what we need, so we can close this issue if there's no intention to support multiple tenants on the rule api.

[sgrzemski]

Hello, My case is: I would like to use Ruler in Grafana Alerting, similarly to: https://community.grafana.com/t/alerting-plugin-with-multi-tenant-mimir-ruler/112040. I have added Mimir as a datasource with federated X-Scope-OrgId header (multiple tenants, separated with a pipe |), but it fails when I want to list alerting rules, because of the very same reason. I don't want to add a datasource for each tenant, so it would be nice to have it enabled. Kind regards, Szymon

@valleedelisle @56quarters

[valleedelisle]

Push your rules on a single tenant and evaluate on multiple sources. After this you can add your alerting data source using only that new single tenant.

[sgrzemski]

Nope, that's awful. I am using grafana-agent to collect prometheus rules across the cluster and set them as rules under a single tenant. That helps me have different rules per different tenants and it's gitops compatibile.

[jenskloster]

We have the same issue.

This must be a bug, since Mimir claims to support Multi Tenancy: https://grafana.com/docs/mimir/latest/references/architecture/components/alertmanager/

[valleedelisle]

I don't think it's a bug, or if it is, it's easily possible to workaround with Federated rule groups. It would be easier if mimir-ruler would support multi-tenancy though.

[craigbryson]

Hi!

We've hit this too and I somewhat agree that its easily worked around with federated rule groups but as far as I can tell there is no way to specify source_tenants[] when configuring rule groups in Grafana Alerting UI? We have a number of users who use the UI to create rules and the rules should be evaluated for multiple tenants - are there any suggestions?

[algo7]

Is there any update on this?

[xdr34m]

Hi, i also bumped into this Problem.

Would be nice if we could specify source_tenants[] in the Rules UI in Grafana and add the tenants we want/are already allowed to see. Or just set all as a standard when configuring for a Datasource that has multiple Tenants?

[CCOLLOT]

I also bumped into this issue. @xdr34m did you manage to specify the source_tenants attribute through a grafana alert (either data source-managed or Grafana-managed)? I was considering using the Grafana operator's CRD to create alert rules but this issue makes me consider moving back to the prometheusRules CRD

[arthurk]

Same for me. It worked fine when testing with a single tenant but then we switched to multi-tenant and it stopped working. I will try to use the federated rule groups as a workaround (difficult because we use PrometheusRule's) but this could be solved in a more user-friendly and consistent way by enabling multi-tenancy in the ruler.

[CCOLLOT]

I ended up creating 1 alert per tenant using loops in the helm templating, and for each alert I had to specify a different folder (the title has to be unique). Now I have about 20 "empty" folders, which is annoying since there is apparently no way to let the users see the alerts but not the folder in the dashboard section. I hope this issue gets fixed in the future

[xdr34m]

We went on and are going full Tenant, with duplicated Metrics. This way everthing is seperated, Not the Best way, but consistent with Loki where there isnt a federationrule.

[vanlongme]

I am on the same page