[Ruler / Query Frontend] Expose the Ruler API via the Query Frontend

[sherifkayad]

Is your feature request related to a problem? Please describe. With recent versions of Grafana and the ability to see Alerts from multiple sources (e.g. Prometheus & Loki) in the frontend, we were getting an error in Grafana Failed to load rules state from Loki: 404 from rule state endpoint. Perhaps ruler API is not enabled?

The answer to Grafana's question is no, we are sure the Ruler API is enabled 😄 .. We have a setup where Loki is running in microservices mode, the Ruler is enabled and configured correctly.

After some digging we found out that the reason of the issue is that the Ruler APIs are missing from the Query Frontend component and since we have the Loki Data Source in Grafana pointing to the Query Frontend.

That was proven simply by launching a test pod and doing the following curl commands:

# hitting the ruler service directly
$ curl http://loki-loki-distributed-ruler.logging:3100/prometheus/api/v1/alerts # => returned 200 ok with a valid response
$ curl http://loki-loki-distributed-ruler.logging:3100/prometheus/api/v1/rules # => returned 200 ok with a valid response
$ curl http://loki-loki-distributed-ruler.logging:3100/loki/api/v1/alerts # => returned 200 ok with a valid response
$ curl http://loki-loki-distributed-ruler.logging:3100/loki/api/v1/rules # => returned 200 ok with a valid response

# hitting the query frontend service
$ curl http://loki-loki-distributed-query-frontend.logging:3100/prometheus/api/v1/alerts
404 page not found
$ curl http://loki-loki-distributed-query-frontend.logging:3100/prometheus/api/v1/rules
404 page not found
$ curl http://loki-loki-distributed-query-frontend.logging:3100/loki/api/v1/alerts
404 page not found
$ curl http://loki-loki-distributed-query-frontend.logging:3100/loki/api/v1/rules
404 page not found

Describe the solution you'd like We would really love to see the rules APIs (/loki/api/v1/rules/*, /loki/api/alerts & /api/prom/rules, /api/prom/alerts as well as their /prometheus counterparts) be part of the Query Frontend APIs. The reason is that I simply think that the Query Frontend should act as the main servant for Grafana and that's now a new extended functionality that Grafana offers with the alerting view.

Describe alternatives you've considered An alternative would be of course using a Gateway Reverse Proxy component (e.g. as per the Helm Chart https://github.com/grafana/helm-charts/tree/main/charts/loki-distributed) to handle the routing to the Ruler whenever needed else route to the Query Frontend .. and of course use that Gateway as the Loki Data Source URL.

I am not sure though if that's the right approach .. The Gateway component for me is a bit unneeded to be honest so I would prefer having the Query Frontend do what the Query Frontend should do; serving Grafana.

Additional context Nothing further.

[uberbrodt]

I am experiencing the same behavior on chart version 0.37.3. Is there a workaround?

[sherifkayad]

@uberbrodt the alternative / workaround is as i described in the issue using the gateway proxy component to reverse proxy not only the query frontend but also the ruler APIs to Grafana.

I would love to see the ruler APIs implemented though as part of the query frontend. I hope we can get the opinion of the project maintainers on that issue.

[parkedwards]

yeah we're hitting the same exact behavior - we're using the loki-distributed helm chart for our loki components as well

200s at the ruler level, but 404s at the gateway level

[timam]

I am facing same issue with loki-distributed. @sherifkayad It will be a great help if you please share your workaround implementation.

[sherifkayad]

@timam @parkedwards As I said we're using the Loki Distrubuted Helm Chart (https://github.com/grafana/helm-charts/tree/main/charts/loki-distributed) .. The workaround is as follows:

• Enable the Gateway Component (Chart Value gateway.enabled set to true)
• I'm using the following Nginx Config (Chart Value gateway.nginxConfig.file):

  # @default -- See values.yaml
  file: |
    worker_processes  5;  ## Default: 1
    error_log  /dev/stderr;
    pid        /tmp/nginx.pid;
    worker_rlimit_nofile 8192;
    events {
      worker_connections  4096;  ## Default: 1024
    }
    http {
      client_body_temp_path /tmp/client_temp;
      proxy_temp_path       /tmp/proxy_temp_path;
      fastcgi_temp_path     /tmp/fastcgi_temp;
      uwsgi_temp_path       /tmp/uwsgi_temp;
      scgi_temp_path        /tmp/scgi_temp;
      default_type application/octet-stream;
      log_format   {{ .Values.gateway.nginxConfig.logFormat }}
      {{- if .Values.gateway.verboseLogging }}
      access_log   /dev/stderr  main;
      {{- else }}
      map $status $loggable {
        ~^[23]  0;
        default 1;
      }
      access_log   /dev/stderr  main  if=$loggable;
      {{- end }}
      sendfile     on;
      tcp_nopush   on;
      resolver {{ .Values.global.dnsService }}.{{ .Values.global.dnsNamespace }}.svc.{{ .Values.global.clusterDomain }};
      {{- with .Values.gateway.nginxConfig.httpSnippet }}
      {{ . | nindent 2 }}
      {{- end }}
      server {
        listen             8080;
        {{- if .Values.gateway.basicAuth.enabled }}
        auth_basic           "Loki";
        auth_basic_user_file /etc/nginx/secrets/.htpasswd;
        {{- end }}
        location = / {
          return 200 'OK';
          auth_basic off;
        }
        location ~ /api/prom/rules.* {
          proxy_pass       http://{{ include "loki.rulerFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
        }
        location ~ /api/prom/alerts.* {
          proxy_pass       http://{{ include "loki.rulerFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
        }
        location = /api/prom/tail {
          proxy_pass       http://{{ include "loki.queryFrontendFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "upgrade";
        }
        location ~ /api/prom/.* {
          proxy_pass       http://{{ include "loki.queryFrontendFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
        }
        location ~ /loki/api/v1/alerts.* {
          proxy_pass       http://{{ include "loki.rulerFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
        }
        location ~ /loki/api/v1/rules.* {
          proxy_pass       http://{{ include "loki.rulerFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
        }
        location = /loki/api/v1/tail {
          proxy_pass       http://{{ include "loki.queryFrontendFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "upgrade";
        }
        location ~ /loki/api/.* {
          proxy_pass       http://{{ include "loki.queryFrontendFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
        }
        location ~ /prometheus/api/v1/alerts.* {
          proxy_pass       http://{{ include "loki.rulerFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
        }
        location ~ /prometheus/api/v1/rules.* {
          proxy_pass       http://{{ include "loki.rulerFullname" . }}.{{ .Release.Namespace }}.svc.{{ .Values.global.clusterDomain }}:3100$request_uri;
        }
        {{- with .Values.gateway.nginxConfig.serverSnippet }}
        {{ . | nindent 4 }}
        {{- end }}
      }
    }

Of course the last piece here is to configure the Grafana Loki Datasource URL to point to the Gateway and not to the Query Frontend.

[timam]

Thanks a lot @sherifkayad

[KeeganOP]

This needs fixing... Every single person that enables grafana 8 alerts and runs loki in distributed mode will run into this and will have to hack the gateway to be a reverse proxy. Seeing as grafana 8 alerts will be enabled by default from grafana 9 onwards, this really should work out the box or at maximum a flag somewhere.

[sherifkayad]

Again wanted to ask if there is any update from the maintainers on this one? @owen-d @slim-bean

[stale[bot]]

Hi! This issue has been automatically marked as stale because it has not had any activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project. A stalebot can be very useful in closing issues in a number of cases; the most common is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, our sincere apologies if you find yourself at the mercy of the stalebot.

[alessioga]

not stale :+1: still happening

[sherifkayad]

Sorry for spamming, but is there any update? @maitainers

[dorkamotorka]

+1

[mwikyo]

+1

[gdidok]

Still happening