Dex OAUTH: "invalid client credentials"

elouanKeryell-Even commented 7 months ago

Checklist:

[x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[x] I've included steps to reproduce the bug.
[x] I've pasted the output of argocd version.

Describe the bug

Authentication through UI often fails with the following error:

failed to get token: oauth2: "invalid client" "invalid client credentials."

(see screenshots further down)

sometimes, opening a new tab, re-typing argocd homepage URL and retrying the login process, makes it work

Our setup:

installation with argocd helm chart
two argocd-server
a dex server
a dex OAUTH connector pointing to our company OAUTH server
a "sticky session" setup using istio

IMPORTANT: something we noticed is the errors seem to occur when interacting with the OLDEST INSTANCE

our two argocd-server instance:

$ kubectl -n argocd-system get pods -l app.kubernetes.io/name=argocd-server
NAME                             READY   STATUS    RESTARTS   AGE
argocd-server-79bc4c4f57-4g5sd   2/2     Running   0          3d
argocd-server-79bc4c4f57-njhwx   2/2     Running   0          12d

dex config (in configmap argocd-cm):

connectors:
  - type: oauth
    id: connector
    name: connector
    config:
      authorizationURL: https://***:443/***/oauth2/multiauth/authorize?acr_values=connectorGROUPE

      claimMapping:
        preferredUsernameKey: sub
        userNameKey: sub

      # Inject sensitive values from secret `argocd-secret`
      # See here: https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/#sensitive-data-and-sso-client-secrets
      clientID: ***
      clientSecret: $dex.config.connector.clientSecret

      redirectURI: https://***/api/dex/callback
      scopes:
        - email
        - openid
        - uid
      tokenURL: https://***:443/***/oauth2/multiauth/access_token
      userIDKey: sub
      userInfoURL: https://***:443/***/oauth2/multiauth/userinfo

istio destination rule to setup sticky session for our multiple argocd-server instances:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: argocd-sticky-session
  namespace: argocd-system
spec:
  host: argocd-server.argocd-system.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpCookie:
          name: user
          ttl: 36000s

To Reproduce

Error does not happen all the time, but here is my process:

open argocd
chose to authenticate through OIDC
get redirected to OIDC provider
log in successfully
get error message on callback to argocd

Expected behavior

I expect to be able to successfully login

Screenshots

Version

$ kubectl -n argocd-system exec -it deployment/argocd-server -- argocd version
argocd: v2.8.4+c279299
  BuildDate: 2023-09-13T19:12:09Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/amd64
FATA[0000] Argo CD server address unspecified
command terminated with exit code 1

Logs

Login failure:

### argocd server
argocd-server-79bc4c4f57-njhwx server time="2024-02-05T09:14:01Z" level=info msg="Performing authorization_code flow login: https://***/api/dex/auth?client_id=argo-cd&redirect_uri=https%3A%2F%2F***%2Fauth%2Fcallback&response_type=code&scope=openid+profile+email+groups&state=***"
argocd-server-79bc4c4f57-njhwx server time="2024-02-05T09:14:11Z" level=info msg="Callback: /auth/callback?code=***&state=***"

### dex server
time="2024-02-05T09:14:11Z" level=info msg="login successful: connector \"connector\", username=\"***\", preferred_username=\"***\", email=\"***@*** (unverified)\", groups=[]"
time="2024-02-05T09:14:11Z" level=info msg="invalid client_secret on token request for client: argo-cd"

Login success:

### argocd server
argocd-server-79bc4c4f57-njhwx server time="2024-02-05T09:39:08Z" level=info msg="Performing authorization_code flow login: https://***/api/dex/auth?client_id=argo-cd&redirect_uri=https%3A%2F%2F***%2Fauth%2Fcallback&response_type=code&scope=openid+profile+email+groups&state=***"
argocd-server-79bc4c4f57-njhwx server time="2024-02-05T09:39:16Z" level=info msg="Callback: /auth/callback?code=***&state=***"
argocd-server-79bc4c4f57-njhwx server time="2024-02-05T09:39:17Z" level=info msg="Web login successful. Claims: {***}"

### dex server
time="2024-02-05T09:39:16Z" level=info msg="login successful: connector \"connector\", username=\"***\", preferred_username=\"***\", email=\"***@*** (unverified)\", groups=[]"

FredNass commented 6 months ago

We are observing the same random behavior with argocd: v2.5.18+b56ef05

msg="invalid client_secret on token request for client: argo-cd" appears for each failed attempt in pod "argocd-dex-server" logs:

time="2024-02-21T15:13:54Z" level=info msg="username \"xxxxxxxxx\" mapped to entry uid=xxxxxxxx,ou=people,dc=xxxxxxxxx,dc=fr"
time="2024-02-21T15:13:54Z" level=info msg="performing ldap search ou=xxxxxxx,dc=xxxxxxxx,dc=fr sub (&(objectClass=groupOfNames)(member=uid=xxxxxxxx,ou=people,dc=xxxxxxxx,dc=fr))"
time="2024-02-21T15:13:54Z" level=info msg="login successful: connector \"ldap\", username=\"firstName lastName" \", preferred_username=\"\", email=\"xxxxxxxxx@xxxxxxxxx.fr\", groups=[\"xxxxxxxxxx\"]"
time="2024-02-21T15:13:54Z" level=info msg="invalid client_secret on token request for client: argo-cd"

Several attempts fail then suddenly one succeeds.

rljohnsn commented 5 months ago

I'm seeing similar experience on v2.9.0+9cf0c69

rljohnsn commented 5 months ago

Session 1

# dex server
time="2024-03-15T10:19:52Z" level=info msg="garbage collection run, delete auth requests=0, auth codes=4, device requests=0, device tokens=0"
time="2024-03-15T10:20:46Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\"]"
time="2024-03-15T10:21:02Z" level=info msg="invalid client_secret on token request for client: argo-cd"
time="2024-03-15T10:21:08Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\"]"
time="2024-03-15T10:21:18Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\"]"
time="2024-03-15T10:21:30Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\"]"
time="2024-03-15T10:21:33Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\"]"
time="2024-03-15T10:21:43Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\"]"
# argo server
time="2024-03-15T10:21:53Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2024-03-15T10:21:53Z" grpc.time_ms=1.113 span.kind=server system=grpc
time="2024-03-15T10:22:00Z" level=info msg="Performing authorization_code flow login: https://argo-mng.eu.postman-alpha.com/api/dex/auth?client_id=argo-cd&redirect_uri=https%3A%2F%2Fargo-mng.eu.postman-alpha.com%2Fauth%2Fcallback&response_type=code&scope=openid+profile+email+groups&state=REDACTED"
2024/03/15 10:22:03 http: proxy error: context canceled
# dex server
time="2024-03-15T10:21:58Z" level=error msg="Failed to authenticate: github: get teams: Get \"https://api.github.com/orgs/postman-eng/members/USER_REDACTED\": context canceled"
time="2024-03-15T10:22:00Z" level=error msg="Failed to authenticate: github: get teams: github: get URL Get \"https://api.github.com/user/teams\": context canceled"
time="2024-03-15T10:22:03Z" level=error msg="Failed to authenticate: github: get teams: github: get URL Get \"https://api.github.com/user/teams\": context canceled"
time="2024-03-15T10:22:15Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\"]"

Session 2

# dex server
time="2024-03-15T12:19:09Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\" \"postman-eng:kubernetes-engineers\" \"postman-eng:ci-cd-engineers\"]"
time="2024-03-15T12:19:30Z" level=info msg="Notifying 1 settings subscribers: [0x4000f7c840]"
time="2024-03-15T12:19:30Z" level=info msg="dex config unmodified"
time="2024-03-15T12:20:39Z" level=error msg="Failed to authenticate: github: get teams: github: get URL Get \"https://api.github.com/user/teams\": context canceled"
time="2024-03-15T12:20:41Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\" \"postman-eng:kubernetes-engineers\" \"postman-eng:ci-cd-engineers\"]"
time="2024-03-15T12:20:51Z" level=error msg="Failed to authenticate: github: get teams: Get \"https://api.github.com/orgs/postman-eng/members/USER_REDACTED\": context canceled"
time="2024-03-15T12:20:54Z" level=info msg="login successful: connector \"github\", username=\"USER_REDACTED\", preferred_username=\"USER_REDACTED\", email=\"USEREMAIL@REDACTED\", groups=[\"postman-eng:developers\" \"postman-eng:kubernetes-engineers\" \"postman-eng:ci-cd-engineers\"]"

ckav370 commented 3 months ago

+1 for this behaviour with Okta SAML and version v2.10.7. Restarting the server deployment seemed to fix

husira commented 1 month ago

we have noticed the same behaviour with v2.11.7. Randomly, it does not work and dex logs the error message: invalid client_secret on token request

argoproj / argo-cd

Dex OAUTH: "invalid client credentials" #17088