grafana / helm-charts

Apache License 2.0
1.67k stars 2.28k forks source link

[loki-distributed] 404 page not found when sending request to /loki/api/v1/rules #3440

Open BeaverBytes opened 18 hours ago

BeaverBytes commented 18 hours ago

Environment

Helm chart configuration

loki:
  config: |
    auth_enabled: false
    ...
    ruler:
      storage:
        type: s3
        s3:
          bucketnames: loki-k3s
          s3forcepathstyle: true
          endpoint: minio.foo.com
          access_key_id: foo
          secret_access_key: bar
          insecure: true
      ring:
        kvstore:
          store: memberlist
      rule_path: /tmp/loki/scratch
      alertmanager_url: http://prometheus-stack-kube-prom-alertmanager.monitoring:9093

  # -- Check https://grafana.com/docs/loki/latest/configuration/#schema_config for more info on how to configure schemas
  schemaConfig:
    configs:
      - from: "2023-01-05"
        store: tsdb
        object_store: s3
        schema: v11
        index:
          prefix: loki_tsdb_index_
          period: 24h
  # -- Check https://grafana.com/docs/loki/latest/configuration/#storage_config for more info on how to configure storages
  storageConfig:
    tsdb_shipper:
      shared_store: s3
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 24h
      resync_interval: 5s
    aws:
      bucketnames: loki-k3s
      s3forcepathstyle: true
      endpoint: minio.foo.com
      access_key_id: bar
      secret_access_key: foo
      insecure: true
serviceMonitor:
  enabled: true
ruler:
  enabled: true
compactor:
  enabled: true

ruler:
  enabled: true
  directories:
    test:
      rules1.yaml: |
        groups:
          - name: error_logs
            rules:
              - alert: MailmanErrorLogsDetected
                expr: |
                  rate({app="mailman"} | json | level="ERROR" | __error__="" [5m])
                for: 1m
                annotations:
                  summary: Mailman error logs detected

gateway:
  ingress:
    enabled: true
    hosts:
      - host: loki-logs.test.com
        paths:
          - path: /
            pathType: Prefix

Issue

When I send a GET request to any of the end ruler endpoints defined here, I get "404 not found". I enabled the ruler component in my helm chart, but that didn't seem to fix my issue. Though, I am able to send request to any of the API endpoints prefixed with /api/prom/rules.

Is it possible that the current app version doesn't support these endpoints? Or could it be related to my helm chart configuration?