argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.84k stars 5.45k forks source link

Persistent SSO login with OpenUnison not working #18735

Closed mattpoel closed 4 months ago

mattpoel commented 4 months ago

Describe the bug

I've configured the OpenUnison SSO login according to documentation, but somehow I keep having a login "loop". Everytime I press the login via openunison button, I see a glimpse of the logged in web page and will directly get forwarded again to login.

Actual login works fine. Example via shell:

# argocd login argo.demo.domain.com --sso
WARN[0001] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
Opening browser for authentication
Performing authorization_code flow login: https://k8sou.demo.domain.com/auth/idp/k8sIdp/auth?access_type=offline&client_id=argocd&code_challenge=asdfasdf&code_challenge_method=S256&redirect_uri=http%3A%2F%2Flocalhost%3A8085%2Fauth%2Fcallback&response_type=code&scope=openid+profile+email+groups&state=yuuBWTNiDPQsXyOZtYygPKtQ
Authentication successful
' ' logged in successfully
Context 'argo.demo.domain.com' updated

Follow up command:

# argocd account get-user-info
Logged In: false

Logs do mention a token verification failure with a tls certificat authority error message, but certificates are signed with Let's Encrypt for OpenUnison and ArgoCD. If I test the mentioned url in my browser, certificate is verified without any issue.

time="2024-06-19T20:08:28Z" level=warning msg="Failed to verify token: failed to verify token: token verification failed for all audiences: error for aud \"argocd\": Failed to query provider \"https://k8sou.demo.domain.com/auth/idp/k8sIdp\": Get \"https://k8sou.demo.domain.com/auth/idp/k8sIdp/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

To Reproduce

Deploy current stable argocd version and configure OpenUnison SSO according to the documentation.

Expected behavior

Login also works via OpenUnison.

Screenshots

login

Version

argocd: v2.11.3+3f344d5
  BuildDate: 2024-06-06T12:31:55Z
  GitCommit: 3f344d54a4e0bbbb4313e1c19cfe1e544b162598
  GitTreeState: clean
  GoVersion: go1.22.4
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.11.3+3f344d5

Logs

time="2024-06-19T20:07:58Z" level=info msg="Initializing OIDC provider (issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp)"
time="2024-06-19T20:07:58Z" level=info msg="OIDC supported scopes: [openid email profile]"
time="2024-06-19T20:07:58Z" level=info msg="Performing authorization_code flow login: https://k8sou.demo.domain.com/auth/idp/k8sIdp/auth?client_id=argocd&redirect_uri=https%3A%2F%2Fargo.demo.domain.com%2Fauth%2Fcallback&response_type=code&scope=openid+profile+email+groups&state=IBiEaHIyvHQcszcZGqCjWRZG"
time="2024-06-19T20:08:23Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:27Z" level=info msg="Callback: /auth/callback?code=LdO3DptaAIDhd%2FFKJHrLZvrB9HpAWWxTTDW9Rffdb6Rk%2Bbdv%2FH%2Ffqu3285bJNFjyyHxCW0p99Jnw01Tvv1YMI7O%2Fvf245f17Ooclz7x8XPN5%2BeOkGkphMEI3CMDj%2BhTfdUIjSsXhold8WQ9SmZ4vzt8ac2baevWzz2v7aAY02JLAKdIcvQRNNbkoGII%2Bwg%2Fbf3V6s562oZP3IwISAJOcnpy9sW8o599CCMhSEa2madR7QVvtyVuBm%2B1iD5sKle8TO9jRejDTU0TPT1FONt%2FXDq86njmBGQV4ghyczmfIoPVq4p1csN6FQmpGFtnZ41FcLsFIPTuEtB5aipDIuNNgvboNccLafdaWzCFXIS0RxefO14s3rtUzHgOFKHXjMrQgdZ17XWjqUaatvgchucao4GmoDAdgW9Qeiw5mVQSDRuShhcnqOG6bfyFnRvTbkUH85CDCUW6P1ASKFHXNEfNqkjtsh1bvHUc8xpa981NH5%2FV6dAMX0OZLW8Z0DIpNJOICqqTf8hlluabO7k3eSWaRUjOrIZ0EIOJREvqkiJW2coCbfmXy7JgrrM%2B%2F35uVtMpkLXjlWoXAKI8OYOMdlwRMcInKea%2F91eqE500xKuHQrYc2h3uFmdrEaGz71RGrIunSrwGv24LhfIzXeMhSoWBkRS5WOC7CO78m%2B%2BwMOPnTg3ozQexLEGJzoNcg3c6Y3hQS55VDmsPDpN%2BU1cAt9rKRWC27Hy3ATpOcJ8dDZEgGfbnRY7EwMcDESu%2B%2BTlT0KMwLhlZT%2BYpmfHtJgJHHBYvOQ4a6b4xVBFIZBgt%2BihRQzJTCd9msJwjCbdwZMTMNeTte7TNZL3dWA6N7fDGHPTj3fBwwWxDWFsN0y5Kg4xj4kFDK5gbrWZGXkkEjbtoS93UQ7heuijtvsBio1S1ch%2F6s0267CGH%2Bt9F%2F%2FwM%3D&state=IBiEaHIyvHQcszcZGqCjWRZG"
time="2024-06-19T20:08:28Z" level=info msg="Web login successful. Claims: {\"aud\":\"argocd\",\"email\":\"\",\"exp\":1718828907,\"groups\":\"eks-administrators\",\"iat\":1718827707,\"iss\":\"https://k8sou.demo.domain.com/auth/idp/k8sIdp\",\"jti\":\"PWQ-ZecQugkQGppQGZJqcQ\",\"name\":\" \",\"nbf\":1718827587,\"nonce\":\"847ccddb-1df7-49e9-bd4d-725b7006ae47\",\"preferred_username\":\"x-54-xx-54-xx-54-xx-49-xx-55-xx-55-xx-50-xx-51-xx-51-xx-48-xfx-52-xdx-56-xbdx-57-xx-53-xx-50-xx-48-xx-52-xcx-56-xa\",\"sub\":\"6661772330f4d8bd95204c8a\"}"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="Initializing OIDC provider (issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp)"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="Initializing OIDC provider (issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp)"
time="2024-06-19T20:08:28Z" level=warning msg="Failed to verify token: failed to verify token: token verification failed for all audiences: error for aud \"argocd\": Failed to query provider \"https://k8sou.demo.domain.com/auth/idp/k8sIdp\": Get \"https://k8sou.demo.domain.com/auth/idp/k8sIdp/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2024-06-19T20:08:28Z" span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=warning msg="Failed to verify token: failed to verify token: token verification failed for all audiences: error for aud \"argocd\": Failed to query provider \"https://k8sou.demo.domain.com/auth/idp/k8sIdp\": Get \"https://k8sou.demo.domain.com/auth/idp/k8sIdp/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="received unary call /version.VersionService/Version" grpc.method=Version grpc.request.content= grpc.service=version.VersionService grpc.start_time="2024-06-19T20:08:28Z" span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2024-06-19T20:08:28Z" grpc.time_ms=20.154 span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2024-06-19T20:08:28Z" grpc.time_ms=26.961 span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="Initializing OIDC provider (issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp)"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="Initializing OIDC provider (issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp)"
time="2024-06-19T20:08:28Z" level=warning msg="Failed to verify token: failed to verify token: token verification failed for all audiences: error for aud \"argocd\": Failed to query provider \"https://k8sou.demo.domain.com/auth/idp/k8sIdp\": Get \"https://k8sou.demo.domain.com/auth/idp/k8sIdp/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="received unary call /version.VersionService/Version" grpc.method=Version grpc.request.content= grpc.service=version.VersionService grpc.start_time="2024-06-19T20:08:28Z" span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Version grpc.service=version.VersionService grpc.start_time="2024-06-19T20:08:28Z" grpc.time_ms=55.763 span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=warning msg="Failed to verify token: failed to verify token: token verification failed for all audiences: error for aud \"argocd\": Failed to query provider \"https://k8sou.demo.domain.com/auth/idp/k8sIdp\": Get \"https://k8sou.demo.domain.com/auth/idp/k8sIdp/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="received unary call /cluster.SettingsService/Get" grpc.method=Get grpc.request.content= grpc.service=cluster.SettingsService grpc.start_time="2024-06-19T20:08:28Z" span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Get grpc.service=cluster.SettingsService grpc.start_time="2024-06-19T20:08:28Z" grpc.time_ms=59.596 span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="Initializing OIDC provider (issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp)"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="Initializing OIDC provider (issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp)"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="Initializing OIDC provider (issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp)"
time="2024-06-19T20:08:28Z" level=warning msg="Failed to verify token: failed to verify token: token verification failed for all audiences: error for aud \"argocd\": Failed to query provider \"https://k8sou.demo.domain.com/auth/idp/k8sIdp\": Get \"https://k8sou.demo.domain.com/auth/idp/k8sIdp/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="received unary call /session.SessionService/GetUserInfo" grpc.method=GetUserInfo grpc.request.content= grpc.service=session.SessionService grpc.start_time="2024-06-19T20:08:28Z" span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=GetUserInfo grpc.service=session.SessionService grpc.start_time="2024-06-19T20:08:28Z" grpc.time_ms=51.134 span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=warning msg="Failed to verify token: failed to verify token: token verification failed for all audiences: error for aud \"argocd\": Failed to query provider \"https://k8sou.demo.domain.com/auth/idp/k8sIdp\": Get \"https://k8sou.demo.domain.com/auth/idp/k8sIdp/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: failed to verify the token" grpc.code=Unauthenticated grpc.method=List grpc.service=application.ApplicationService grpc.start_time="2024-06-19T20:08:28Z" grpc.time_ms=28.85 span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=warning msg="Failed to verify token: failed to verify token: token verification failed for all audiences: error for aud \"argocd\": Failed to query provider \"https://k8sou.demo.domain.com/auth/idp/k8sIdp\": Get \"https://k8sou.demo.domain.com/auth/idp/k8sIdp/.well-known/openid-configuration\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="finished unary call with code Unauthenticated" error="rpc error: code = Unauthenticated desc = invalid session: failed to verify the token" grpc.code=Unauthenticated grpc.method=List grpc.service=cluster.ClusterService grpc.start_time="2024-06-19T20:08:28Z" grpc.time_ms=36.75 span.kind=server system=grpc
time="2024-06-19T20:08:28Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
time="2024-06-19T20:08:28Z" level=info msg="Initializing OIDC provider (issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp)"
christianh814 commented 4 months ago

@mattpoel Are you using direct OIDC setup or are you doing it with Dex? (Judging from the output I assume it's OIDC directly but want to make sure)

mattpoel commented 4 months ago

@christianh814 my trust configuration looks like the following:

apiVersion: openunison.tremolo.io/v1
kind: Trust
metadata:
  name: argocd
  namespace: openunison
spec:
  accessTokenSkewMillis: 120000
  accessTokenTimeToLive: 1200000
  authChainName: login-service
  clientId: argocd
  codeLastMileKeyName: lastmile-oidc
  codeTokenSkewMilis: 60000
  publicEndpoint: true
  redirectURI:
  - https://argo.demo.domain.com/auth/callback
  - http://localhost:8085/auth/callback
  signedUserInfo: true
  verifyRedirect: true

and the configmap for the OIDC configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  url: https://argo.demo.domain.com
  oidc.config: |-
    name: OpenUnison
    issuer: https://k8sou.demo.domain.com/auth/idp/k8sIdp
    clientID: argocd
    requestedScopes: ["openid", "profile", "email", "groups"]
agaudreault commented 4 months ago

Usually, tls: failed to verify certificate: x509: certificate signed by unknown authority errors indicates that https://k8sou.demo.domain.com/auth/idp/k8sIdp/.well-known/openid-configuration does not have a valid trusted certificate. You can try to reproduce with curl from a container running where argo-cd is deployed. If you need to add --insecure, then you will either need a certificate trusted by a known CA, or add the root CA to the trust store.

https://argo-cd.readthedocs.io/en/stable/operator-manual/declarative-setup/#repositories-using-self-signed-tls-certificates-or-are-signed-by-custom-ca might be a way that works for argo. It is documented for repository, but since the server is making the call to the oidc provider, I think it might work too.

mattpoel commented 4 months ago

@agaudreault this was also my thinking, but I already spun up a throwaway netshoot pod in the argocd namespace to check on that, and the certificate/request to the URL was fine :neutral_face:

I now thought, a couple of restarts and re-applying the configuration can't do any harm and by:

:boom: I was in (because I had already signed on in OpenUnison).

I guess the restarts of the argocd-server pod fixed this. According to documentation the configuration can be done without any restarts, but in case somebody might end up with the same problem, give it a try :wink:

Thanks for your support! @agaudreault @christianh814

Kampe commented 4 months ago

I too am running into this with my sso provider as I use self signed certs I assume there's either a way to tell Dex to not verify or I guess just mount the crt.

agaudreault commented 4 months ago

Glad to hear you were able to fix it. A restart of the repo-server might be necessary. It would seem that the configs are not always updated, depending on what changes

Might be related to https://github.com/argoproj/argo-cd/issues/18576 for the SSO config reload.