Allow specifying remote_write tenant in ruler

[arch-xtof]

Describe the bug I am trying to create a recording rule for tenant main in loki and sent the metric results to tenant 0 in Mimir. Even though I specify the necessary headers for this to happen, Loki still pushes metrics to main tenant in Mimir.

To Reproduce Steps to reproduce the behavior:

1. Started Loki Ruler 2.8.4
2. Create recording rule for tenant X, that pushes metrics to tenant Y in Mimir

Expected behavior Metrics go to the Mimir tenant that I specify in X-Scope-OrgID header

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Ansible, directly applying manifests.

Screenshots, Promtail config, or terminal output Here is my Loki ruler config, I have tried all combinations with/without quotes for headers name and value and lowercase/uppercase for headers name

    ruler:
      remote_write:
        enabled: true
        clients:
          mimir:
            url: http://mimir-nginx.mimir.svc.cluster.local/api/v1/push
            headers:
              X-Scope-OrgID: "0"

[dvrkni]

hello, same here, no matter what i put the value not overriding. but the syntax seems ok cause im able to inject custom header and in other hand it works on plain prometheus instance. can X-Scope-OrgID header be immutable by mistake?

[dvrkni]

seems solved by adding authorization on loki to enable multi-tenancy and configuring required tenant id on grafana agent log instance.

apiVersion: monitoring.grafana.com/v1alpha1
kind: LogsInstance
metadata:
  name: primary
spec:
  clients:
    - url: "http://loki-gateway/loki/api/v1/push"
      tenantId: requiredId

and loki

sidecar:
  rules:
    folder: /loki/rules/requiredId
loki:
  rulerConfig:
    wal:
      dir: /tmp/ruler-wal
    storage:
      type: local
      local:
        directory: /loki/rules
    rule_path: /tmp/scratch
    alertmanager_url: http://mimir-nginx/alertmanager
    remote_write:
      enabled: true
      clients:
        mimir:
          url: http://mimir-nginx/api/v1/push
    ring:
      kvstore:
        store: inmemory
    enable_api: true

[javiplx]

As you're using mimir-nginx, you might workaround this issue until remote-write config gets fixed by tweaking nginx configuration. There is a block

      # Ensure that X-Scope-OrgID is always present, default to the no_auth_tenant for backwards compatibility when multi-tenancy was turned off.
      map $http_x_scope_orgid $ensured_x_scope_orgid {
        default $http_x_scope_orgid;
        "" "anonymous";
      }

that you can use to do a kind of tenant mapping

[arch-xtof]

Thank you for the suggestions, I already worked around this by sending metrics to one of the Prometheus federators instead of Mimir.

[gespi1]

I am running a loki-distributed chart ( 0.78.3) , running a multi-tenant setup, and we have a loki recording rules on several tenants. If we were to setup something like

ruler:
  remote_write:
    enabled: true
    clients:
      tenant1:
        url: http://mimir-nginx.mimir.svc.cluster.local/api/v1/push
        headers:
          X-Scope-OrgID: tenant1
      tenant2:
        url: http://mimir-nginx.mimir.svc.cluster.local/api/v1/push
        headers:
          X-Scope-OrgID: tenant2

How would the ruler know what recording rules belong to what tenant and know what client to use? Would it send the recording rules results to the correct tenant ?

EDIT: nevermind we can do it per-tenant basis on the overrides ruler_remote_write_config