How to set up ingress for grafana and prometheus?

[lenaxia]

What did you do? While I know I can do port forwarding, I'd prefer to have TOBS behind a reverse proxy like all my other services. I use Traefik and tried to configure grafana like I do normally:

kind: HelmRelease
metadata:
  name: tobs
  namespace: monitoring
spec:
  chart:
    spec:
      version: "12.0.1"
  values:
    grafana:
      ingress:
        enabled: true
        ingressClassName: "traefik"
        annotations:
          #cert-manager.io/cluster-issuer: "letsencrypt-staging"
          cert-manager.io/cluster-issuer: "ca-issuer" # self signed dev cert
          traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
        hosts:
          - &host "grafana.${SECRET_DEV_DOMAIN}"
        tls:
          - hosts:
              - "grafana.${SECRET_DEV_DOMAIN}"
            secretName: "grafana-devdomain-tls-staging"

Did you expect to see some different?

This should've resulted in an ingress object being created in the monitoring namespace and a cert request going through. I did not see either of these and grafana remained unavailable.

Environment

K3s being deployed using fluxcd2

• tobs version:

  tobs version: 12.0.1

kind: HelmRelease
metadata:
  name: tobs
  namespace: monitoring
  labels:
    app: tobs
spec:
  interval: 15m
  chart:
    spec:
      chart: tobs
      version: 12.0.1
      sourceRef:
        kind: HelmRepository
        name: timescale-charts
        namespace: flux-system
      interval: 15m
  install:
    createNamespace: true
    remediation:
      retries: 5
  upgrade:
    remediation:
      retries: 5
        #  values:

• Kubernetes version information:

  kubectl version

Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7+k3s1", GitCommit:"ac70570999c566ac3507d2cc17369bb0629c1cc0", GitTreeState:"clean", BuildDate:"2021-11-29T16:40:13Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

• Kubernetes cluster kind:

  Bootstrapped through fluxcd

Anything else we need to know?:

[nhudson]

This should've resulted in an ingress object being created in the monitoring namespace and a cert request going through. I did not see either of these and grafana remained unavailable.

Can you be a bit more descriptive? I see the annotations being added to the Grafana configuration, but are they being rendered and applied as an actual Deployment object once it's applied to the cluster? Do you see an Ingress object on the cluster?

From what I can tell there isn't a ingressClassName setting in the kube-prometheus-stack configuration. This is what I am looking at https://github.com/prometheus-community/helm-charts/blob/85bfdf4f72dea1327ab52d3b4d508f7f5bd633ae/charts/kube-prometheus-stack/values.yaml#L730-L766

Also it took me a few minutes, but I think you have your override values incorrectly specified. If you're looking to override the Grafana defaults you will need to have a root of kube-prometheus-stack

  values:
    kube-prometheus-stack:
      grafana:
        ingress:
          enabled: true
          annotations:
            #cert-manager.io/cluster-issuer: "letsencrypt-staging"
            cert-manager.io/cluster-issuer: "ca-issuer" # self signed dev cert
            traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
          hosts:
            - &host "grafana.${SECRET_DEV_DOMAIN}"
          tls:
            - hosts:
                - "grafana.${SECRET_DEV_DOMAIN}"
              secretName: "grafana-devdomain-tls-staging"

It also might be easier to put these into a file and reference them with valuesFrom: instead of listing the values over multiple lines in the HelmRelease object configuration you have. Either should work, it's just what I would do instead of listing them all freely.

In the end I don't think this is an issue with tobs itself, but an issue with configuration on the Traefik or FluxCD side. When I set kube-prometheus-stack.grafana.ingress.enabled=true, it will render the Ingress CR needed to set the object in the cluster

kube-prometheus-stack:
  grafana:
    enabled: true
    ingress:
      enabled: true
      annotations:
        cert-manager.io/cluster-issuer: "ca-issuer" # self signed dev cert
        traefik.ingress.kubernetes.io/router.entrypoints: "websecure"

---
# Source: tobs/charts/kube-prometheus-stack/charts/grafana/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: release-name-grafana
  namespace: default
  labels:
    helm.sh/chart: grafana-6.32.10
    app.kubernetes.io/name: grafana
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/version: "9.0.7"
    app.kubernetes.io/managed-by: Helm
  annotations:
    cert-manager.io/cluster-issuer: "ca-issuer"
    traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
spec:
  rules:
    - http:
        paths:
          - backend:
              service:
                name: release-name-grafana
                port:
                  number: 80
            path: /
            pathType: Prefix

[nhudson]

@lenaxia I am going to close this. If you feel like your questions were not answered please re-open! Thanks!

[lenaxia]

Thanks nhudson! That got it working, appreciate the help. Now having other issues with no data coming in, digging in on that now.