Configuring OIDC with Dex causes weave-gitops to crash

[kjhadd]

Describe the bug Configuring OIDC with Dex and Google/Github causes the weave-gitops to go into a crashloop.

Using HelmRelease for both weave-gitops and dex.

Essentially following the guide here https://docs.gitops.weave.works/docs/guides/setting-up-dex/. Also attempted using the Google connector instead of the Github connector in Dex, with the same result.

weave-gitops crashes when the oidc-auth secret exists with the client configuration defined in the staticClients section of Dex Config. It does however not crash if instead the oidc-auth client credentials are set directly to credentials issued by Google.

Environment

• Weave-Gitops Version v0.28.0
• Dex Version v2.37.0 (also tried v2.31.0)
• Flux Version v0.41.2
• Kubernetes version v1.26.5-gke.1400 TLS with Traefik ingress controller and cert-manager.

To Reproduce Steps to reproduce the behavior:

• Deploy Dex from HelmRelease (see config below)
• Create oidc-auth secret with client configuration as defined in Dex config
• Deploy Gitops from HelmRelease (see config below)

Expected behavior

I access the weave-gitops frontend, press the login with OIDC button and get redirected through Dex to login with Google/Github.

Actual Behavior

Pod crashes on init with the following logs, and goes into a crashloop.

ww-gitops-6c8c9dbcb9-8f9xt 2023-08-23T06:03:17.933Z    INFO    gitops    cmd/cmd.go:135    Version    {"version": "v0.28.0", "git-commit": "3ad24e71", "branch": "HEAD", "buildtime": "2023-07-19_18:14:46"}
ww-gitops-6c8c9dbcb9-8f9xt I0823 06:03:18.988755       1 request.go:690] Waited for 1.010939416s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/networking.gke.io/v1alpha1?timeout=32s
ww-gitops-6c8c9dbcb9-8f9xt 2023-08-23T06:03:19.244Z    DEBUG    gitops    auth/init.go:17    Registering authentication methods    {"methods": ["oidc"]}
ww-gitops-6c8c9dbcb9-8f9xt 2023-08-23T06:03:19.249Z    DEBUG    gitops    auth/init.go:49    OIDC config    {"IssuerURL": "https://dex.mycluster.mydomain.com", "ClientID": "weave-gitops", "ClientSecretLength": **, "RedirectURL": "https://gitops.mycluster.mydomain.com/oauth2/callback", "TokenDuration": "1h0m0s"}
Stream closed EOF for flux-system/ww-gitops-6c8c9dbcb9-8f9xt (weave-gitops)

Additional Context (screenshots, logs, etc)

dex-helmrelease.yaml

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: dex
  namespace: dex
spec:
  interval: 5m
  chart:
    spec:
      chart: dex
      version: 0.15.3
      sourceRef:
        kind: HelmRepository
        name: dex
        namespace: dex
      interval: 1m
  values:
    fullnameOverride: "dex"
    image:
      tag: v2.37.0
    envVars:
    - name: GOOGLE_CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: google-client
          key: client-id
    - name: GOOGLE_CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: google-client
          key: client-secret
    - name: GITHUB_CLIENT_ID
      valueFrom:
        secretKeyRef:
          name: github-client
          key: client-id
    - name: GITHUB_CLIENT_SECRET
      valueFrom:
        secretKeyRef:
          name: github-client
          key: client-secret
    config:
      # Set it to a valid URL
      issuer: https://dex.mycluster.mydomain.com

      logger:
        level: debug

      # See https://dexidp.io/docs/storage/ for more options
      storage:
        type: kubernetes
        config:
          inCluster: true

      staticClients:
      - name: 'Weave GitOps Core'
        id: weave-gitops
        secret: supersecret
        redirectURIs:
        - 'https://gitops.mycluster.mydomain.com/oauth2/callback'
        - 'http://0.0.0.0:9001/oauth2/callback'
        - 'http://localhost:4567/oauth2/callback'
        - 'https://localhost:4567/oauth2/callback'
        - 'http://localhost:3000/oauth2/callback'

      connectors:
      - type: github
        id: github
        name: GitHub
        config:
          clientID: $GITHUB_CLIENT_ID
          clientSecret: $GITHUB_CLIENT_SECRET
          redirectURI: https://dex.mycluster.mydomain.com/callback
          orgs:
          - name: my-github-org
            teams:
            - my-team
      - type: oidc
        id: google
        name: Google
        config:
          issuer: https://accounts.google.com
          clientID: $GOOGLE_CLIENT_ID
          clientSecret: $GOOGLE_CLIENT_SECRET
          redirectURI: https://dex.mycluster.mydomain.com/callback

    ingress: 
      enabled: false  # Ingress configured outside of HelmRelease

weave-gitops oidc config in k8s secret

kubectl create secret generic oidc-auth \
--namespace flux-system \
--from-literal=issuerURL='https://dex.mycluster.mydomain.com' \
--from-literal=clientID='weave-gitops' \
--from-literal=clientSecret='supersecret' \
--from-literal=redirectURL='https://gitops.mycluster.mydomain.com/oauth2/callback' \
--from-literal=tokenDuration='1h' \
--from-literal=customScopes=email,profile

weave-gitops-helmrelease.yaml

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  annotations:
    metadata.weave.works/description: This is the Weave GitOps Dashboard.  It provides
      a simple way to get insights into your GitOps workloads.
  name: ww-gitops
  namespace: flux-system
spec:
  chart:
    spec:
      chart: weave-gitops
      version: 4.0.26 # helm chart to 4.0.26 to use gitops 0.28.0
      sourceRef:
        kind: HelmRepository
        name: ww-gitops
  interval: 1h0m0s
  values:
    fullnameOverride: "ww-gitops"
    logLevel: debug
    rbac:
      create: true
      viewSecretsResourceNames: ["oidc-auth"]
    additionalArgs:
      - --auth-methods
      - oidc
    image:
      tag: "v0.28.0"

[bigkevmcd]

hey @kjhadd I've had a look at this.

While I accept that it definitely shouldn't crash, it's not clear what's going on.

I've tried reproducing it, and invalid credentials for Dex are correctly reported in the UI.

From Dex

time="2023-08-24T14:04:58Z" level=info msg="invalid client_secret on token request for client: example-app"

And Weave GitOps

dev-weave-gi… │ 2023-08-24T14:04:58.958Z    ERROR   gitops.auth-server  auth/server.go:341  failed to exchange auth code for token  {"code": "<snip>", "error": "oauth2: cannot fetch token: 401 Unauthorized\nResponse: {\"error\":\"invalid_client\",\"error_description\":\"Invalid client credentials.\"}"}

Do you have any further logs at all?

[kjhadd]

Since Weave Gitops pod crashes on init I am never able to run an auth request, so it is all the logs I have. Dex pod has nothing related in the logs.

Do you have any proposal for logs I should look for? During registration of clients, is there anything that Weave Gitops does that should leave a mark in my kubernetes logs somewhere?

[bigkevmcd]

@kjhadd I'm looking through the code for any cases where we would just terminate.

How about -p in kubectl logs do you have any more information there?

[bigkevmcd]

Hey @kjhadd I think this should be fixed, we bumped a dependency which I think was the cause of this one https://github.com/weaveworks/weave-gitops/pull/3968

[bigkevmcd]

I'm going to close this, if it's not fixed, please feel free to reopen.

[evandam]

Just a heads up @kjhadd I thought I was running into this too, but I realized the issuer URL for my Dex service wasn't actually accessible inside my cluster.

Not sure if it's the same case for you, but it's possible a call to Dex is hanging when the server is starting.