OIDC google account stopped working after helm uninstall / install: {"error": "http: named cookie not present"}

[schmidp]

Describe the bug

I have GitOps deployed with flux and integrated it with google OIDC and it worked fine. I was able to login with my google account and watch flux do its thing. Then I was restructuring my flux repo layout and moved the GitOps deployment from one Kustomization to another. So basically flux did a helm uninstall and then an install, if I am correct. Also the namespace stayed the same.

But after the reinstallation I can no longer login via OIDC. The helm chart and all its config values stayed the same.

In the browser I get the following message:

{"message":"failed to verify ID token: failed to verify signature: fetching keys oidc: get keys failed: 403 Forbidden \u003c!DOCTYPE html\u003e\n\u003chtml lang=en\u003e\n  \u003cmeta charset=utf-8\u003e\n  \u003cmeta name=viewport content=\"initial-scale=1, minimum-scale=1, width=device-width\"\u003e\n  \u003ctitle\u003eError 403 (Forbidden)!!1\u003c/title\u003e\n  \u003cstyle\u003e\n    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* \u003e body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}\n  \u003c/style\u003e\n  \u003ca href=//www.google.com/\u003e\u003cspan id=logo aria-label=Google\u003e\u003c/span\u003e\u003c/a\u003e\n  \u003cp\u003e\u003cb\u003e403.\u003c/b\u003e \u003cins\u003eThat’s an error.\u003c/ins\u003e\n  \u003cp\u003eYour client does not have permission to get URL \u003ccode\u003e/oauth2/v3/certs\u003c/code\u003e from this server.  \u003cins\u003eThat’s all we know.\u003c/ins\u003e\n","code":500}

In the logs I can see:

│ 2023-07-14T08:14:39.117Z    ERROR    gitops.auth-server    auth/server.go:456    failed to get ID Token cookie from request    {"error": "http: named cookie not present"}                                                                                  │
│ 2023-07-14T08:14:39.117Z    INFO    gitops    middleware/middleware.go:61    request error    {"uri": "/oauth2/userinfo", "status": 400}                                                                                                                    │
│ 2023-07-14T08:14:46.673Z    INFO    gitops    middleware/middleware.go:65    server error    {"uri": "/oauth2/callback?state=XXXX%3D&code=XXXXX&scope=email+profile+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.profile+openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&authuser=0 │
│ &hd=XXXX&prompt=none", "status": 500}  

Helm:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  annotations:
    metadata.weave.works/description: This is the source location for the Weave GitOps
      Dashboard's helm chart.
  labels:
    app.kubernetes.io/component: ui
    app.kubernetes.io/created-by: weave-gitops-cli
    app.kubernetes.io/name: weave-gitops-dashboard
    app.kubernetes.io/part-of: weave-gitops
  name: ww-gitops
  namespace: weave-gitops
spec:
  interval: 1h0m0s
  type: oci
  url: oci://ghcr.io/weaveworks/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  annotations:
    metadata.weave.works/description: This is the Weave GitOps Dashboard.  It provides
      a simple way to get insights into your GitOps workloads.
  name: ww-gitops
  namespace: weave-gitops
spec:
  install:
    createNamespace: true
  targetNamespace: weave-gitops
  releaseName: weave-gitops
  chart:
    spec:
      chart: weave-gitops
      sourceRef:
        kind: HelmRepository
        name: ww-gitops
  interval: 1h0m0s
  values:
    adminUser:
      create: true
      # Backup admin user password hash
      passwordHash: XXXX
      username: admin
    envVars:
      - name: WEAVE_GITOPS_FEATURE_OIDC_BUTTON_LABEL
        value: "Login with Google Account"
    ingress:
      enabled: true
      className: "nginx-internal"
      annotations:
        cert-manager.io/cluster-issuer: letsencrypt-prod-dns
        nginx.ingress.kubernetes.io/use-regex: "true"
        external-dns.alpha.kubernetes.io/hostname: bla.com
        external-dns.alpha.kubernetes.io/access: private
      hosts:
        - host: bla.com
          paths:
            - path: "/"
              pathType: ImplementationSpecific
      tls:
        - secretName: ww-gitops-tls
          hosts:
            - bla.com

Environment

• Weave-Gitops Version 0.27 (4.0.25 chart)
• Flux Version 2.0.0
• Kubernetes version 1.27

To Reproduce Steps to reproduce the behavior:

see above.

Expected behavior

I can login with OIDC.

Actual Behavior

I get the error message above.

Additional Context (screenshots, logs, etc)

[z15alall]

Hi @schmidp,

I think this is same problem like mentionned in this issue : #3783

Indeed cookie received (id_token) from your oidc provider are greater thant 4096 char (contain too much, roles, groups...) I think authentification management was changed and now cookie are store in the web browser.

If id_token cookie are greater than 4096 char webrowser remove this cookie from request to respect RFC that why the error message from weave-gitops is : {"error": "http: named cookie not present"}

Cookie size in webbrowser are limited to 4096 char.

[bigkevmcd]

We are working on a fix for this (replacing the direct cookie with a Session token instead).

[bigkevmcd]

@schmidp Have you set the OIDC verifier URL to https://accounts.google.com ?

[schmidp]

@schmidp Have you set the OIDC verifier URL to https://accounts.google.com ?

yes

clientID: zzzz.apps.googleusercontent.com                                                                                                                                                                          
clientSecret: yyy                                                                                                                                                                                                          
customScopes: email,profile                                                                                                                                                                                                                                 
issuerURL: https://accounts.google.com                                                                                                                                                                                                                      
redirectURL: https://xxxx.com/oauth2/callback   

[bigkevmcd]

@schmidp The error message in the text above is Your client does not have permission to get URL /oauth2/v3/certs is the client id and secret correct?

[schmidp]

@bigkevmcd Yes, I have just rechecked it and its correct. Also it worked before I did the helm uninstall/install without changing the k8s secret in between.

[bigkevmcd]

@schmidp We are working to fix an issue where if the OIDC ID token results in cookie that is bigger than 4096 bytes, the browser drops it and we get that http: named cookie not present.

But this behaviour hasn't changed recently, could it be the case that you have a lot of groups which would be pushing the cookie beyond this size?

[schmidp]

@schmidp We are working to fix an issue where if the OIDC ID token results in cookie that is bigger than 4096 bytes, the browser drops it and we get that http: named cookie not present.

But this behaviour hasn't changed recently, could it be the case that you have a lot of groups which would be pushing the cookie beyond this size?

Yes that is possible, I have a lot of groups. Maybe the helm install/uninstall is just a coincidence.

[schmidp]

So today the problem resolved itself. I swear by my liver, I did not change any config and no redeployment. We also did not change any groups or added removed me from groups. So either the helm install/uninstall changed something that was cached somewhere or google changed something or its just magic.

[bigkevmcd]

@schmidp Thanks for letting us know, we will push on with the session implementation which would solve large groups cookies, but otherwise, seems like it's working for you.

[kfalconer]

I had the same issue just happen on our deployment. It had been running for about 17 days, and started getting the same cookie not present error. Restarted the deployment and it went away.

[bigkevmcd]

We have now released support for session storage, which removes the ID token from the cookie.

https://github.com/weaveworks/weave-gitops/releases/tag/v0.31.2

I'm going to close this now, but feel free to reopen if it's not working.