unable to validate rule group payload

Describe the bug Sample prometheusrules from this repo doesn't work.

To Reproduce Steps to reproduce the behavior:

Deploy loki through distributed helm chart.

Set prometheusRule.enable to true i.e.

prometheusRule:
enabled: true
namespace: loki
labels:
alloy: loki
groups:
- name: loki_rules
  rules:
    - expr: |-
        histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m]))
        by (le, cluster, job))
      record: cluster_job:loki_request_duration_seconds:99quantile
      labels:
        loki: true
      annotations:
        loki: true

Grafana alloy operator tries to create the rule but receives an error from the loki ruler:

could not parse expression for record 'cluster_job:loki_request_duration_seconds:99quantile' in group 'loki_rules': parse error at line 1, col 1: syntax error: unexpected IDENTIFIER

grafana-alloy config

loki.rules.kubernetes "alloyrules" {
address = "http://loki-loki-distributed-ruler.loki.svc.cluster.local:3100"
rule_selector {
match_labels = {
  alloy = "loki",
}
}
}

loki ruler configuration:

ruler:
  storage:
    type: gcs
    gcs:
      bucket_name: dev-loki-ruler
  ring:
    kvstore:
      store: memberlist
  rule_path: /tmp/loki/scratch
  alertmanager_url: http://mimir-nginx.mimir.svc.cluster.local/alertmanager
  external_url: http://mimir-nginx.mimir.svc.cluster.local/alertmanager
  enable_api: true
  enable_alertmanager_v2: true
  remote_write:
    enabled: true
    clients:
      fake:
        url: http://mimir-nginx.mimir.svc.cluster.local/api/v1/push
    add_org_id_header: true
  wal:
    dir: /tmp/loki/ruler-wal

Expected behavior The rule gets created in the loki ruler properly.

Additionally, I am confused that metric concerning loki is stored in mimir. How does the loki ruler know how to query data out of loki or mimir depending on the query??

Environment:

Infrastructure: GKE
Deployment tool: helm distributed chart

grafana / loki

unable to validate rule group payload #14775