Ruler pass x-org-scopeid/org id/tenant ID to alertmanager

[kovaxur]

Is your feature request related to a problem? Please describe. As I understand, ruler now is not adding the x-org-scopeid header to the alertmanager API calls, when it sends an alert. This would be nice if someone also uses Cortex with a tenant-aware alertmanager setup.

Describe the solution you'd like Add the tenant ID of the tenant, on which ruler executes the rules. This would also work, when tenants (auth) is not enabled with the "fake" user.

Describe alternatives you've considered I don't have any other ideas now.

Additional context Currently with an alertmanager cluster set up by a tenant-aware Cortex we receive:

level=error ts=2022-04-22T17:25:30.961461794Z caller=notifier.go:527 user=REMOVED_TENANT_ID alertmanager=http://cortex-alertmanager.cortex.svc.cluster.local:8080/api/v1/alerts count=1 msg="Error sending alert" err="bad response status 400 Bad Request"

[gavinwoolley]

@kovaxur Did you manage to get this working somehow? I have just implemented Loki Ruler and trying to get it to fire Alerts with MIMIR alertmanager. I am having the same problem, the Loki ruler does not send the tenantId, so the MIMIR alertmanager wont accept the payload.

[kovaxur]

@gavinwoolley, just installed a new instance of alertmanager separately for loki

[gavinwoolley]

Thanks for replying. Sorry I don't follow how another alert manager solves the issue of Loki not passing tenant ID to alert manager? Could you elaborate please

[shaarif-khan]

@gavinwoolley were you able to find the solution, we are also facing the same issue.

[gavinwoolley]

it turns out, it just worked, it does pass the tenantID from rules created in grafana UI, that have the loki datasource set to the same tenantID.

this is loki ruler config i use

    ruler:
      storage:
        type: s3
        s3:
          insecure: false
          sse_encryption: true
          http_config:
            insecure_skip_verify: true
      ring:
        kvstore:
          store: memberlist
          prefix: rulers/
      rule_path: /tmp/loki/scratch
      enable_alertmanager_v2: true
      alertmanager_url: http://mimir-alertmanager-headless.mimir.svc.cluster.local:8080/alertmanager
      remote_write:
        clients:
          mimir_distributor:
            url: http://mimir-distributor-headless.mimir.svc.cluster.local:8080/api/v1/push
        enabled: true
      wal:
        dir: /tmp/loki/ruler-wal