tenant-setting service is not reachable if not using ingress for routing

[ecktom]

Describe the bug

We did add https://github.com/grafana/explore-profiles to our Grafana instance and also brought up tenant-setting and adhoc-profiles by using the latest version of the pyroscope helm chart which did re-add them as mentioned in https://github.com/grafana/pyroscope/issues/3463#issuecomment-2268419795.

We're running pyroscope in microservice mode without the ingress resources as we're not exposing it externally, this means alloy and Grafana do connect to the services directly. In the case of Grafana we do connect to the query-frontend on http://grafana-pyroscope-query-frontend.pyroscope.svc.cluster.local:4040. We're however ending up receiving a 404 on the settings page of "Explore Profiles" which also make some features unavailable.

Looking at other Grafana components, they mostly ship with a gateway which is responsible for those routing aspects. There is however no gateway component provided for Pyroscope. Is my assumption correct, that using the https://github.com/grafana/explore-profiles plugin currently does not work within the setup mentioned above?

Environment

• Grafana 11.1.3
• explore-profiles 0.1.5
• Pyroscope 1.7.1

[ecktom]

I confirmed this by copy/pasting the gateway part from Loki and adjusting it to Pyroscope (see below). After deploying and adjusting the datasource to look at http://pyroscope-gateway.pyroscope.svc.cluster.local:80 everything works fine

While working on it, I noticed that the path definitions in here and here do not seem to be correct. They apparently need to be prefixed with /pyroscope

# This config is taken from Loki Gateway and just adjusted some locations to match Pyroscope
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: pyroscope-gateway
  namespace: pyroscope
  labels:
    app.kubernetes.io/name: pyroscope
    app.kubernetes.io/instance: grafana-pyroscope
    app.kubernetes.io/component: gateway
data:
  nginx.conf: |-
    worker_processes  5;  ## Default: 1
    error_log  /dev/stderr;
    pid        /tmp/nginx.pid;
    worker_rlimit_nofile 8192;

    events {
      worker_connections  4096;  ## Default: 1024
    }

    http {
      client_body_temp_path /tmp/client_temp;
      proxy_temp_path       /tmp/proxy_temp_path;
      fastcgi_temp_path     /tmp/fastcgi_temp;
      uwsgi_temp_path       /tmp/uwsgi_temp;
      scgi_temp_path        /tmp/scgi_temp;

      client_max_body_size  4M;

      proxy_read_timeout    600; ## 10 minutes
      proxy_send_timeout    600;
      proxy_connect_timeout 600;

      proxy_http_version    1.1;

      default_type application/octet-stream;
      log_format  main '$remote_addr - $remote_user [$time_local]  $status '
                  '"$request" $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

      access_log   /dev/stderr  main;

      sendfile     on;
      tcp_nopush   on;

      resolver kube-dns.kube-system.svc.cluster.local.;

      server {
        listen             8080;
        listen             [::]:8080;

        # Query Frontend
        location = / {
          proxy_pass       http://grafana-pyroscope-query-frontend.pyroscope.svc.cluster.local:4040$request_uri;  
        }

        location = /pyroscope/render-diff {
          proxy_pass       http://grafana-pyroscope-query-frontend.pyroscope.svc.cluster.local:4040$request_uri; 
        }
        location ^~ /pyroscope/render-diff/ {
          proxy_pass       http://grafana-pyroscope-query-frontend.pyroscope.svc.cluster.local:4040$request_uri; 
        }

        location = /pyroscope/render {
          proxy_pass       http://grafana-pyroscope-query-frontend.pyroscope.svc.cluster.local:4040$request_uri; 
        }
        location ^~ /pyroscope/render/ {
          proxy_pass       http://grafana-pyroscope-query-frontend.pyroscope.svc.cluster.local:4040$request_uri; 
        }

        location = /querier.v1.QuerierService {
          proxy_pass       http://grafana-pyroscope-query-frontend.pyroscope.svc.cluster.local:4040$request_uri; 
        }
        location ^~ /querier.v1.QuerierService/ {
          proxy_pass       http://grafana-pyroscope-query-frontend.pyroscope.svc.cluster.local:4040$request_uri; 
        }

        # Settings
        location = /settings.v1.SettingsService {
          proxy_pass       http://grafana-pyroscope-tenant-settings.pyroscope.svc.cluster.local:4040$request_uri;
        }
        location ^~ /settings.v1.SettingsService/ {
          proxy_pass       http://grafana-pyroscope-tenant-settings.pyroscope.svc.cluster.local:4040$request_uri;
        }

        # Ad-hoc
        location = /adhocprofiles.v1.AdHocProfileService {
          proxy_pass       http://grafana-pyroscope-ad-hoc-profiles.pyroscope.svc.cluster.local:4040$request_uri;
        }
        location ^~ /adhocprofiles.v1.AdHocProfileService/ {
          proxy_pass       http://grafana-pyroscope-ad-hoc-profiles.pyroscope.svc.cluster.local:4040$request_uri;
        }
      }
    }
---
apiVersion: v1
kind: Service
metadata:
  name: pyroscope-gateway
  namespace: pyroscope
  labels:
    app.kubernetes.io/name: pyroscope
    app.kubernetes.io/instance: grafana-pyroscope
    app.kubernetes.io/component: gateway
  annotations:
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 80
      targetPort: http
      protocol: TCP
  selector:
    app.kubernetes.io/name: pyroscope
    app.kubernetes.io/instance: grafana-pyroscope
    app.kubernetes.io/component: gateway
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pyroscope-gateway
  namespace: pyroscope
  labels:
    app.kubernetes.io/name: pyroscope
    app.kubernetes.io/instance: grafana-pyroscope
    app.kubernetes.io/component: gateway
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/name: pyroscope
      app.kubernetes.io/instance: grafana-pyroscope
      app.kubernetes.io/component: gateway
  template:
    metadata:
      labels:
        app.kubernetes.io/name: pyroscope
        app.kubernetes.io/instance: grafana-pyroscope
        app.kubernetes.io/component: gateway
    spec:
      serviceAccountName: pyroscope
      enableServiceLinks: true
      priorityClassName: p6
      securityContext:
        fsGroup: 101
        runAsGroup: 101
        runAsNonRoot: true
        runAsUser: 101
      terminationGracePeriodSeconds: 30
      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:1.23-alpine
          imagePullPolicy: IfNotPresent
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          readinessProbe:
            httpGet:
              path: /
              port: http
            initialDelaySeconds: 15
            timeoutSeconds: 1
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            readOnlyRootFilesystem: true
          volumeMounts:
            - name: config
              mountPath: /etc/nginx
            - name: tmp
              mountPath: /tmp
            - name: docker-entrypoint-d-override
              mountPath: /docker-entrypoint.d
          resources:
            limits:
              memory: 100Mi
            requests:
              cpu: 200m
              memory: 100Mi
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: pyroscope
                    app.kubernetes.io/instance: grafana-pyroscope
                    app.kubernetes.io/component: gateway
                topologyKey: kubernetes.io/hostname
      volumes:
        - name: config
          configMap:
            name: pyroscope-gateway
        - name: tmp
          emptyDir: {}
        - name: docker-entrypoint-d-override
          emptyDir: {}