grafana to many redirects during login if 2 grafana pods running

[AlexABorisov]

I have issue with running running 2 grafana pods. Issue related to many redirects during login. Browser get this message and back to login page. Issue not reproducible if only 1 grafana pod is run. Grafana run on openshift and via nginx ingress. Grafana version Grafana v8.3.3 (30bb7a93ca) Helm chart version 8.3.4 configuration of ingress

kind: Ingress
metadata:
  annotations:
    meta.helm.sh/release-name: oce-system-kube-prometheus-stack
    meta.helm.sh/release-namespace: oce-system
  creationTimestamp: "2023-06-30T08:58:20Z"
  generation: 4
  labels:
    app.kubernetes.io/instance: oce-system-kube-prometheus-stack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 8.3.3
    helm.sh/chart: grafana-6.20.5
  name: oce-system-kube-prometheus-stack-grafana
  namespace: oce-system
  resourceVersion: "2947154"
  uid: 2744ee01-676f-48d5-aec8-7a90a469fe3a
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - backend:
          service:
            name: oce-system-kube-prometheus-stack-grafana
            port:
              number: 3000
        path: /grafana
        pathType: Prefix
status:
  loadBalancer:
    ingress:
    - ip: 172.30.142.7

grafana ini

 grafana.ini: |
    [analytics]
    check_for_updates = true
    [auth.anonymous]
    enabled = false
    org_role = Viewer
    [grafana_net]
    url = https://grafana.net
    [log]
    mode = console
    [panels]
    disable_sanitize_html = true
    [paths]
    data = /var/lib/grafana/data
    logs = /var/log/grafana
    plugins = /var/lib/grafana/plugins
    provisioning = /etc/grafana/provisioning
    [server]
    domain = localhost
    root_url = %(protocol)s://%(domain)s/grafana
    serve_from_sub_path = true

No issues with single pod deployment

[kstevensonnv]

Could you let me know if you resolved this @AlexABorisov?

I'm experiencing the same issue running on EKS. Grafana is exposed via an Application Load Balancer and accessed at: https://my.grafana.instance.com

If one pod is running I can log in successfully. If two pods are running it redirect loops until the browser limit is exhausted. If three pods are running I am redirected straight back to the login page.

grafana.ini:

[analytics]
check_for_updates = true
[auth.generic_oauth]
allow_sign_up = true
api_url = https://$user_pool_id.auth.$region.amazoncognito.com/oauth2/userInfo
auth_url = https://$user_pool_id.auth.$region.amazoncognito.com/oauth2/authorize
client_id = $__file{/etc/secrets/grafana-auth-generic-oauth/client-id}
client_secret = $__file{/etc/secrets/grafana-auth-generic-oauth/client-secret}
enabled = true
name = Cognito
role_attribute_path = ("cognito:groups" | contains([*], 'grafana-admin') && 'Admin' || contains([*], 'grafana-viewer') && 'Viewer' )
role_attribute_strict = true
scopes = email openid phone profile
token_url = https://$user_pool_id.auth.$region.amazoncognito.com/oauth2/token
use_refresh_token = true
[grafana_net]
url = https://grafana.net
[log]
level = debug
mode = console
[paths]
data = /var/lib/grafana/
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning
[server]
domain = grafana.$grafana_domain
root_url = https://%(domain)s

New pods, no persistence configured:

NAME                                                        READY   STATUS    RESTARTS   AGE
kube-prometheus-stack-grafana-74755f6f44-4nftc              3/3     Running   0          60s
kube-prometheus-stack-grafana-74755f6f44-znx5r              3/3     Running   0          19s

Accessing Grafana using a new private window in Firefox. No previous cookies or cache.

Logs from pod 1:

logger=oauth.generic_oauth t=2023-10-27T04:12:32.778388439Z level=debug msg="Getting user info"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.778414308Z level=debug msg="Extracting user info from OAuth token"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.778545991Z level=debug msg="Received id_token" raw_json="{\"at_hash\":\"$hash\",\"sub\":\"$sub\",\"cognito:groups\":[\"grafana-admin\",\"$group1\",\"$group2\",\"$group3\"],\"email_verified\":false,\"iss\":\"https:\\/\\/cognito-idp.$region.amazonaws.com\\/$user_pool_id\",\"cognito:username\":\"$username\",\"origin_jti\":\"$origin_jti\",\"aud\":\"$aud\",\"event_id\":\"776e6ccc-2e24-499e-b942-e2cca8056344\",\"token_use\":\"id\",\"auth_time\":1698379952,\"exp\":1698383552,\"iat\":1698379952,\"jti\":\"$jti\",\"email\":\"$email\"}" data="Name: , Displayname: , Login: , Username: , Email: $email, Upn: , Attributes: map[]"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.778560488Z level=debug msg="Getting user info from API"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.925457121Z level=debug msg="HTTP GET" url=https://$user_pool_id.auth.$region.amazoncognito.com/oauth2/userInfo status="200 OK" response_body="{\"sub\":\"$sub\",\"email_verified\":\"false\",\"email\":\"$email\",\"username\":\"$username\"}"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.925508939Z level=debug msg="Received user info response from API" raw_json="{\"sub\":\"$sub\",\"email_verified\":\"false\",\"email\":\"$email\",\"username\":\"$username\"}" data="Name: , Displayname: , Login: , Username: $username, Email: $email, Upn: , Attributes: map[]"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.925606042Z level=debug msg="Processing external user info" source=token data="Name: , Displayname: , Login: , Username: , Email: $email, Upn: , Attributes: map[]"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.925684281Z level=debug msg="Unable to find user info name"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.925760369Z level=debug msg="Set user info email from extracted email" email=$email
logger=oauth.generic_oauth t=2023-10-27T04:12:32.926056898Z level=debug msg="Processing external user info" source=API data="Name: , Displayname: , Login: , Username: $username, Email: $email, Upn: , Attributes: map[]"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.926074142Z level=debug msg="Unable to find user info name"
logger=oauth.generic_oauth t=2023-10-27T04:12:32.926078925Z level=debug msg="Setting user info login from username field" username=$username
logger=oauth.generic_oauth t=2023-10-27T04:12:32.926084189Z level=debug msg="User info result" result="Id: $sub, Name: , Email: $email, Login: $username, Role: Admin, Groups: []"
logger=org.sync t=2023-10-27T04:12:32.957791025Z level=debug msg="Syncing organization roles" id=user:2 extOrgRoles=map[1:Admin]
logger=accesscontrol.service t=2023-10-27T04:12:32.964994559Z level=debug msg="fetch permissions from store" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:32.966900101Z level=debug msg="cache permissions" key=rbac-permissions-1-user-2
logger=auth t=2023-10-27T04:12:32.972019433Z level=debug msg="user auth token created" tokenId=1 userId=2 clientIP=$ip userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/118.0" authToken=$authtoken
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:32.972182091Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=$ip time_ms=301 duration=301.332729ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login/:name
logger=auth t=2023-10-27T04:12:33.061279596Z level=debug msg="seen token" tokenId=1 userId=2 clientIP=$ip userAgent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/118.0" authToken=$authtoken
logger=accesscontrol.service t=2023-10-27T04:12:33.062577921Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.083598987Z level=debug msg="fetch permissions from store" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.085459217Z level=debug msg="cache permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.09496504Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:33.095055536Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=39 duration=39.229532ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:33.178195722Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.180914191Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.190815477Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:33.190894986Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.449817ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:33.272744725Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.282288946Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.291932272Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:33.292019344Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=21 duration=21.059788ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:33.376804604Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.379331904Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.389316584Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:33.389407953Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.454128ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:33.469653826Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.471800257Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.481454069Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:33.481544383Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=13 duration=13.604411ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:33.564043188Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.566552542Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.585550913Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:33.590022108Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=27 duration=27.968403ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:33.671936728Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.674571669Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.684638388Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:33.684804575Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.597763ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:33.768997979Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.772323378Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.781781182Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:33.781875866Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.602692ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:33.870741154Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.873166469Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:33.882769629Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:33.882851317Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=13 duration=13.831416ms size=24 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:38.878931424Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:38.881242685Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:38.890777437Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:39.636019484Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:39.636207696Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=$ip time_ms=1 duration=1.955175ms size=0 referer= handler=/api/live/ws
logger=live t=2023-10-27T04:12:39.670772091Z level=debug msg="Client connected" user=2 client=$client
logger=accesscontrol.service t=2023-10-27T04:12:39.680515047Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:39.774832473Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=avatar t=2023-10-27T04:12:39.775042003Z level=debug msg="avatar.fetch(fetch new avatar)" url=https://secure.gravatar.com/avatar/13f48e1bd0108fc46b0fe5bcc0f103c7?
logger=avatar t=2023-10-27T04:12:39.775180359Z level=debug msg="Fetching avatar url with parameters" url="https://secure.gravatar.com/avatar/13f48e1bd0108fc46b0fe5bcc0f103c7?d=retro&r=pg&size=200"
logger=accesscontrol.service t=2023-10-27T04:12:39.788336697Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=avatar t=2023-10-27T04:12:39.823841146Z level=debug msg="Fetching avatar url with parameters" url="https://secure.gravatar.com/avatar/13f48e1bd0108fc46b0fe5bcc0f103c7?d=404"
logger=accesscontrol.service t=2023-10-27T04:12:39.831619336Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=ngalert.scheduler t=2023-10-27T04:12:40.001744815Z level=debug msg="Alert rules fetched" rulesCount=0 foldersCount=0 updatedRules=0
logger=ngalert.state.manager t=2023-10-27T04:12:40.183809218Z level=debug msg="Recording state cache metrics" now=2023-10-27T04:12:40.183800978Z
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-10-27T04:12:40.225135299Z level=debug msg="Start walking disk" path=/tmp/dashboards
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-10-27T04:12:43.750900024Z level=debug msg="Start walking disk" path=/tmp/dashboards
logger=live t=2023-10-27T04:12:44.803921461Z level=debug msg="Client disconnected" user=2 client=$client reason="connection closed" elapsed=5.133115387s
logger=accesscontrol.service t=2023-10-27T04:12:44.818765325Z level=debug msg="fetch permissions from store" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:44.820233798Z level=debug msg="cache permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:44.822371757Z level=debug msg="fetch permissions from store" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:44.823843501Z level=debug msg="cache permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:44.833366914Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.471728147Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.542093198Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.555653005Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.557112928Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.579808084Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.592463725Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.59709677Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.623209553Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.66956177Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.67234936Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.682012409Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:45.682115553Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.341098ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:45.751730138Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.754450433Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.764509233Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:45.764676482Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.667091ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:45.83461654Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.83669182Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.846588447Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:45.84669053Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=13 duration=13.919946ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:45.917888201Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.920025244Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:45.930241204Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:45.930417112Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.24024ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:45.998676956Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.001464153Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.011919545Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:46.012102842Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=15 duration=15.190222ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:46.082743118Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.085651386Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.095429386Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:46.095529103Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.532792ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:46.165981264Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.16834058Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.178467528Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:46.178557884Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.365681ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:46.248146426Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.250251851Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.260128329Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:46.260716619Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.202862ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=accesscontrol.service t=2023-10-27T04:12:46.331635894Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.333448225Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:46.334340946Z level=info msg="Request Completed" method=GET path=/api/live/ws status=-1 remote_addr=$ip time_ms=2 duration=2.480404ms size=0 referer= handler=/api/live/ws
logger=accesscontrol.service t=2023-10-27T04:12:46.33661618Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.347542272Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:46.347627975Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=17 duration=17.895853ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=live t=2023-10-27T04:12:46.365530937Z level=debug msg="Client connected" user=2 client=bd183b1b-ab6e-4781-9f18-4b6f524f1610
logger=accesscontrol.service t=2023-10-27T04:12:46.420368326Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.423200166Z level=debug msg="using cached permissions" key=rbac-permissions-0-user-2
logger=accesscontrol.service t=2023-10-27T04:12:46.433152325Z level=debug msg="using cached permissions" key=rbac-permissions-1-user-2
logger=context userId=2 orgId=1 uname=$username t=2023-10-27T04:12:46.433350163Z level=info msg="Request Completed" method=GET path=/login status=302 remote_addr=$ip time_ms=14 duration=14.684441ms size=32 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/login
logger=live t=2023-10-27T04:12:46.50128685Z level=debug msg="Client disconnected" user=2 client=bd183b1b-ab6e-4781-9f18-4b6f524f1610 reason="connection closed" elapsed=135.730195ms

Logs from pod 2:

logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:21.085878832Z level=info msg="Request Completed" method=GET path=/login/generic_oauth status=302 remote_addr=$ip time_ms=0 duration=94.511µs size=369 referer=https://$grafana_domain/login handler=/login/:name
logger=provisioning.dashboard type=file name=sidecarProvider t=2023-10-27T04:12:24.093194384Z level=debug msg="Start walking disk" path=/tmp/dashboards
logger=ngalert.scheduler t=2023-10-27T04:12:30.002021275Z level=debug msg="Alert rules fetched" rulesCount=0 foldersCount=0 updatedRules=0
logger=authn.service t=2023-10-27T04:12:33.01422995Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.014430415Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=874.126µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:33.135129394Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.135318311Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=808.159µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:33.231172008Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.231280007Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=585.16µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:33.334287523Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.33446604Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=750.278µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:33.429323695Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.429423529Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=654.749µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:33.521129694Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.521208179Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=717.025µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:33.630088703Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.630161485Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=674.44µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:33.727723663Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.727830171Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=667.497µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:33.825796041Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.825977453Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=679.26µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:33.927637852Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:33.927706947Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=647.712µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=ngalert.state.manager t=2023-10-27T04:12:35.936150263Z level=debug msg="Recording state cache metrics" now=2023-10-27T04:12:35.936141275Z
logger=authn.service t=2023-10-27T04:12:39.643557967Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:39.643627638Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:39.64370602Z level=info msg="Request Completed" method=GET path=/api/dashboards/home status=401 remote_addr=$ip time_ms=0 duration=563.472µs size=40 referer=https://$grafana_domain/ handler=/api/dashboards/home
logger=authn.service t=2023-10-27T04:12:39.716292316Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:39.716350141Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:39.716545026Z level=info msg="Request Completed" method=GET path=/api/dashboards/home status=401 remote_addr=$ip time_ms=0 duration=815.912µs size=40 referer=https://$grafana_domain/ handler=/api/dashboards/home
logger=authn.service t=2023-10-27T04:12:39.787870072Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=avatar t=2023-10-27T04:12:39.789083133Z level=debug msg="avatar.fetch(fetch new avatar)" url=https://secure.gravatar.com/avatar/13f48e1bd0108fc46b0fe5bcc0f103c7?
logger=avatar t=2023-10-27T04:12:39.789376979Z level=debug msg="Fetching avatar url with parameters" url="https://secure.gravatar.com/avatar/13f48e1bd0108fc46b0fe5bcc0f103c7?d=retro&r=pg&size=200"
logger=authn.service t=2023-10-27T04:12:39.790952886Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:39.791027269Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:39.791563362Z level=info msg="Request Completed" method=GET path=/api/search status=401 remote_addr=$ip time_ms=1 duration=1.042704ms size=40 referer=https://$grafana_domain/ handler=/api/search/
logger=avatar t=2023-10-27T04:12:39.843232923Z level=debug msg="Fetching avatar url with parameters" url="https://secure.gravatar.com/avatar/13f48e1bd0108fc46b0fe5bcc0f103c7?d=404"
logger=authn.service t=2023-10-27T04:12:39.866187377Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:39.866261595Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:39.866422683Z level=info msg="Request Completed" method=GET path=/api/search status=401 remote_addr=$ip time_ms=0 duration=816.271µs size=40 referer=https://$grafana_domain/ handler=/api/search/
logger=ngalert.scheduler t=2023-10-27T04:12:40.003574244Z level=debug msg="Alert rules fetched" rulesCount=0 foldersCount=0 updatedRules=0
logger=authn.service t=2023-10-27T04:12:45.506995807Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.507199735Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.507686705Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=$ip time_ms=1 duration=1.287083ms size=40 referer= handler=/api/live/ws
logger=authn.service t=2023-10-27T04:12:45.527017583Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=authn.service t=2023-10-27T04:12:45.543023731Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.543080002Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.543120895Z level=info msg="Request Completed" method=GET path=/api/search status=401 remote_addr=$ip time_ms=1 duration=1.827707ms size=40 referer="https://$grafana_domain/?orgId=1" handler=/api/search/
logger=authn.service t=2023-10-27T04:12:45.558412728Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.558457994Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.558506628Z level=info msg="Request Completed" method=GET path=/api/plugins status=401 remote_addr=$ip time_ms=0 duration=454.533µs size=40 referer="https://$grafana_domain/?orgId=1" handler=/api/plugins
logger=authn.service t=2023-10-27T04:12:45.558656909Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.558680568Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.558703155Z level=info msg="Request Completed" method=GET path=/api/search status=401 remote_addr=$ip time_ms=1 duration=1.020154ms size=40 referer="https://$grafana_domain/?orgId=1" handler=/api/search/
logger=authn.service t=2023-10-27T04:12:45.588802709Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.588856661Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.588897795Z level=info msg="Request Completed" method=GET path=/api/search status=401 remote_addr=$ip time_ms=0 duration=622.17µs size=40 referer="https://$grafana_domain/?orgId=1" handler=/api/search/
logger=authn.service t=2023-10-27T04:12:45.596931569Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.59700722Z level=info msg="Request Completed" method=GET path=/api/login/ping status=401 remote_addr=$ip time_ms=0 duration=550.912µs size=26 referer="https://$grafana_domain/?orgId=1" handler=/api/login/ping
logger=authn.service t=2023-10-27T04:12:45.614618628Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.614669836Z level=warn msg=Unauthorized error="user token not found" remote_addr=$ip traceID=
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.614707882Z level=info msg="Request Completed" method=GET path=/api/search status=401 remote_addr=$ip time_ms=0 duration=583.42µs size=40 referer="https://$grafana_domain/?orgId=1" handler=/api/search/
logger=authn.service t=2023-10-27T04:12:45.633375062Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.633447736Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=610.767µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:45.716247553Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.716362589Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=696.408µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:45.799390047Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.799468898Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=673.132µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:45.881327555Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.881436758Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=662.245µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:45.964081323Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:45.964246394Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=724.487µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:46.046544814Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:46.046621583Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=670.943µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:46.129808639Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:46.129883107Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=673.039µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:46.212429661Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:46.212614725Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=719.096µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:46.294204881Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:46.294274876Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=649.851µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:46.385407068Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:46.38557797Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=823.408µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/
logger=authn.service t=2023-10-27T04:12:46.467167386Z level=warn msg="Failed to authenticate request" client=auth.client.session error="user token not found"
logger=context userId=0 orgId=0 uname= t=2023-10-27T04:12:46.467241389Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=$ip time_ms=0 duration=648.98µs size=29 referer=https://$user_pool_id.auth.$region.amazoncognito.com/ handler=/

[ssrahul96]

+1,

facing similar issue with 2 replicas and Azure AD

few of my analysis,

1. when 2 pods are configured,

the azure ad provider redirects to https://{{host}}/login/azuread?code=xxxxxxx&state=xxxxxx&session_state=xxxxx, which inturn redirects to /grafana/

2. when 1 pod is configured,

the azure ad provider redirects to https://{{host}}/login/azuread?code=xxxxxxx&state=xxxxxx&session_state=xxxxx, which inturn redirects to /

May be this stackoverflow thread could be a workaround, yet to try

[ssrahul96]

Folks,

Try adding persistence

deploymentStrategy:
  type: Recreate
persistence:
  enabled: true

The issue is resolved for me

[Adrian43211234]

I solved it with this sticky adjustment to my Grafana IngressRoute (not part of helm values)

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`xxxxxxxxxx`) && PathPrefix(`/grafana`)
      kind: Rule
      services:
        - name: grafana
          port: 80
          sticky:
            cookie:
              httpOnly: true
              name: cookie
              secure: true
              sameSite: none

This ends up like this:

Honestly I am not deep into this problem now. Can this also be fixed on the service? https://kubernetes.io/docs/reference/networking/virtual-ips/#session-affinity

Can I run with this change?