[kube-prometheus-stack] redirect to the login page after successful login

[MichaelKora]

Describe the bug a clear and concise description of what the bug is.

I have kube-prometheus-stack running on my Kubernetes cluster. When using Port Forwarding i can login into the Grafana dashboard with my creds. but when trying to do the same using the generated FQDN (created with the ingress: config of the the helm chart), i get redirected to the login page after seeing "Logged In".

I've seen some opened issue here and there but nothing helped so far. So i would truly appreciate your help.

What's your helm version?

version.BuildInfo{Version:"v3.14.0", GitCommit:"3fc9f4b2638e76f26739cd77c7017139be81d0ea", GitTreeState:"clean", GoVersion:"go1.21.5"}

What's your kubectl version?

Client Version: v1.29.1 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.11

Which chart?

prometheus-community/kube-prometheus-stack

What's the chart version?

61.1.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

grafana:
   grafana.ini:
     users:
       viewers_can_edit: false
       login_default_org_id: 1
     server:
       serve_from_sub_path: true
     security:
       allow_embedding: true
       cookie_samesite: lax
       cookie_secure: false
     paths:
       data: /var/lib/grafana/
       logs: /var/log/grafana
       plugins: /var/lib/grafana/plugins
       provisioning: /etc/grafana/provisioning
     analytics:
       check_for_plugin_updates: true
       check_for_updates: true

  ingress:
    enabled: true
    ingressClassName: traefik
    annotations:
      kubernetes.io/ingress.allow-http: "true"
    hosts:
      - my-grafana.mydomain.com
    path: /
    pathType: ImplementationSpecific
    tls: []

Enter the command that you execute and failing/misfunctioning.

helm upgrade -i -n monitoring prometheus prometheus-community/kube-prometheus-stack -f values.yaml

Anything else we need to know?

No response

[zeritti]

If you set an ingress host, you should set root_url (defaults). The default Grafana's config otherwise produces http://localhost:3000. Something like this:

grafana:
  grafana.ini:
    server:
      domain: my-grafana.mydomain.com
      protocol: http
      http_port: 80
      root_url: %(protocol)s://%(domain)s:%(http_port)s/

Edit: correction of port variable Edit: I forgot that domain gets set in the default Grafana's values if ingress has been set. It need not be set then.

[MichaelKora]

Hey @zeritti. thanks for the response. i saw that root_url is also being set in the defaults

anywho, i updated the values to the following:

grafana:
   grafana_ini:
      server: 
        protocol: http
        http_port: 3000
        domain: my-grafana.example.com

and this has not impacted the behavior. i am still getting redirected to the login page. Also using Port 80 as you recommended returned this error: Error: ✗ *api.HTTPServer run error: failed to open listener on address 0.0.0.0:80: listen tcp 0.0.0.0:80: bind: permission denied

[MichaelKora]

a redirect is automatically performed on every link with the following pattern my-grafana.example.com/<xxxx>.... everything with suffix in the link is redirected to my-grafana.example.com/login

[zeritti]

root_url is a URL by which Grafana points to itself on its pages (like an external URL). According to your ingress for Grafana, you do not use https and I reckon your ingress controller listens on 80/tcp. Setting http_port to 3000 will not work unless your ingress controller listens on the same port.

Your root_url as "full public facing url" should in the end give http://my-grafana.mydomain.com. Give it a go:

grafana:
  grafana.ini:
    server:
      root_url: http://my-grafana.mydomain.com

[MichaelKora]

it is still not working and this is my current config:

grafana:
   grafana.ini:
    users:
      viewers_can_edit: false
      login_default_org_id: 1
    server:
      root_url: http://my-grafana.mydomain.com
      serve_from_sub_path: false
      http_port: 3000
    auth:
      disable_login_form: false
      disable_signout_menu: false
    security:
      allow_embedding: true
      cookie_samesite: lax
      cookie_secure: false
      angular_support_enabled: false

using http_port: 80 resulted in a permission denied Error as mentioned before

[MichaelKora]

Setting http_port to 3000 will not work unless your ingress controller listens on the same port.

If that was the case i wont be able to access the UI at all..i can access the UI using the FQDN. but i just seem to be looping back to the login page. my guesses

1- the chart converts any URL with a suffix to /login..because entering http://my-grafana.mydomain.com/testblabla does not return a page not found error but redirects me to http://my-grafana.mydomain.com/login page instead. and i can indeed see on the dev mode of the browser that a redirect is excuted...

2- the cookies are not handled properly. there my be a config about that, that am missing...

but that is just me trying to make sense out of the current situation

[zeritti]

Setting http_port to 3000 will not work unless your ingress controller listens on the same port.

Apologies, this was wrong, the port is of course the internal Grafana's port, it need not normally be changed. I wanted to say that that port does not occur in root_url where one uses mostly 80 or 443.

[MichaelKora]

right. the issue has to be somewhere else. and am having a hard time finding it out