Argo CD UI getting logged out in 5 mins while integrated with keycloak

[preethignanesh]

Issue We have installed Argo CD of version 2.5.3. We have integrated Argo CD with keycloak following the steps in https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/keycloak/#integrating-keycloak-and-argocd With this integration when I login to the Argo CD UI via keycloak, then I see that the UI logs out the user in 5 mins which is the access token lifespan timeout value.

Steps to reproduce -

• Install Argo CD and integrate with keycloak
• Login to Argo CD via keycloak

After 5 mins, UI logs out even if the user is in the middle of doing something.

Expected behavior If a token is expired, use the refresh token to refresh an existing token. We would like to have the user not getting logged out in 5 mins.

[IvanovOleg]

I have the same problem with 2.5.9 server.oidc.cache.expiration: 30m doesn't help most likely it is related to Access Token Lifespan parameter in the keycloak which is equal to 5 minutes by default

[IvanovOleg]

It looks like argocd is making a login attempt and then refresh attempt right after each time token expires. I believe it should make a token refresh request only and this is a bug. Increasing access token lifespan on the keycloak side helps to reduce amount of relogins, but it's an ugly fix.

[azlydnev]

I added:

oidc.config: |
  skipAudienceCheckWhenTokenHasNoAudience: true

and restart pods argocd-server-* seems its helped me Use Keycloak manual from https://argo-cd.readthedocs.io/en/stable/operator-manual/user-management/keycloak/ Argo: 2.6.0 server logs on each page refresh: time="2023-02-08T14:37:56Z" level=warning msg="Failed to verify token: token is expired"

[IvanovOleg]

@azlydnev I don't like that ArgoCD when token is expired tries to reload the page twice. First time it reloads a page and displays "failed to load data", then it makes another refresh and displays data. It's annoying.

[preethignanesh]

From a customer point of view its annoying if the page has to load twice due to refresh.

[leandro-alt]

I'm facing the same problem. Would be great a solution for this. Is this issue related to https://github.com/argoproj/argo-cd/issues/10126 ?

[Volevanius]

Same here. Did anyone find workaround?

[ricardojdsilva87]

Hello everyone, we are experiencing a similar issue, basically in the UI we are redirected several times to relogin again. It seems that argo detects that the session token expired but instead of requesting a new one and keep on the same argoCD page it start over the entire oauth workflow (with 2nd factor authentication enabled). This can happen right after 1 minute or take 30 minutes to happen, but it happens in random pages.

We did some changes recently on the deployment to try and reduce the times this situation occurs: Our oidc configuration is as follows:

            oidc.config: |
              name: OIDC name
              issuer: https://oidc.url
              clientID: $oidc-secret:oidc.clientID
              clientSecret: $oidc-secret:oidc.clientSecret
              requestedScopes: ["openid"]

Also saw this change on v2.6.3 that seems to do some changes on the oidc refresh tokens so we upgraded to the latest available version (v2.6.4).

Have the affinity enabled on the service and also on the ingress:

          ## Server service configuration
          service:
            # -- Used to maintain session affinity. Supports `ClientIP` and `None`
            sessionAffinity: "ClientIP"

          ingress:
            # -- Enable an ingress resource for the Argo CD server
            enabled: true
            # -- Additional ingress annotations
            annotations:
              nginx.ingress.kubernetes.io/affinity: cookie
              nginx.ingress.kubernetes.io/affinity-mode: persistent

We have HPA enabled for the server pods, but there is not much rotation of these (they can be up for days), so we expect that the token would persist and keep the user session information and prevent the reload.

Also instead of the login token being saven on each server pod is there any possibility in thinking of storing them in the redis database? This way we would have a central place where to store these (Currently we are using a HA external redis instance connected to argo).

Thanks for your help!

[adrian-sturm]

@ricardojdsilva87 So did updating argocd to v2.6.4 and adding the affinity-options resolve the issue for you? We are currently on v2.5.2 and it seems to us that the refresh token is not used at all...

[ricardojdsilva87]

Hello @adrian-sturm unfortunately no, we still have thee same issue, the UI reloads randomly and goes trought all the OIDC steps (it's automatic and we do not need to do any input, the page just refreshes completely).

We usually tend to keep argocd up to date with the latest version, so I'll keep checking if this issue is fixed.

Also the sessionAffinity option set on nginx didn't have any effect, what is strange is that we have the same nginx ingress with the nginx.ingress.kubernetes.io/affinity annotations set and it works fine for other tools that have multiple replicas, on some of them we have an expiring session timeout that is when the OIDC token expires and relogin is needed.

I would say that this needs to be changed/fixed on argocd core. I don't have enough experience in go to to contribute :), but I can test different configurations to check if the issue persists.

Thanks!

[hbollon]

We're experiencing the same issue with the exact same setup on our side. The refresh token is apparently not used and argocd just replay the whole login workflow when the id token expire. It's really annoying while we're using tokens with short lifetime (5 minutes)...

Our setup:

• ArgoCD v2.7.2 (in K8S)
• Keycloak v21.1.0 (in AWS ECS)

[Hronom]

Please, same issue, even with 1 hour token it's very disruptive

[valentin-stefan-popa]

Any news on this? I have same problem in ArgoCd v2.6.3. I authenticate ArgoCD with keycloak and randomly at 5-10 minutes I get some errors regarding token and need to re authenticate

[jo-krk]

Same here, any news on this?

[AlessandroBravo]

Same here using ArgoCD v2.8.4

[byrneo]

Having the exact same problem also. Running ArgoCD v2.8.4 and SSO re-auths every ~5 minutes.

[amirdamirov]

I increased "Access Token Lifespan" in keycloak it helped.

[rouke-broersma]

I'm also seeing this with Authentik oidc since a couple weeks.

[jack-kearney]

I'm seeing this as well — has there been any progress on a solution here?

[emostov]

+1 on the issue

[andrewkmin]

Bump, would appreciate some updates. Thanks!

[SalehBorhani]

still have problem!

[kzgrzendek]

Same here, it would be great if we could a=have any news about a potential fix, thanks :)

[SalehBorhani]

I fixed my problem with setting time for Access Token Lifespan. It was Never expires before but it didn't work.

[Hronom]

Problem here is that Argo CD can't renew token in background, basically it not support OAuth 2 flow properly

[jpark0910]

Is changing the Access Token lifespan the proposed solution or just a workaround for this issue? Is anyone working on the fix in the code?

[rouke-broersma]

Is changing the Access Token lifespan the proposed solution or just a workaround for this issue? Is anyone working on the fix in the code?

It's a workaround not a fix, it's not secure.

[ChristianCiach]

I'm pretty sure the issue lies here:

https://github.com/argoproj/argo-cd/blob/8ca96a9d0e9f26821e72a163c5630a25f34ee1ce/util/session/sessionmanager.go#L543-L547

It makes no sense to return an error to the UI just because the access-token is invalid, since the refresh-token will most likely still be valid and a simple token-refresh would most likely succeed. The server should try to refresh the tokens at this point and only return an error to the client if the refresh fails.

Edit: In fact, Argo-CD doesn't seem to support refresh-tokens at all. The oidc.go does cache the access-token, but it does not cache the refresh-token and there is no function to support token-refresh. The whole code seems to be based on the legacy "implicit flow" that should not be used nowadays.

When starting the server with ARGOCD_SSO_DEBUG=1, the UI displays a "redeem refresh token"-button, but this, too, is broken (at least when not using the "implicit flow").

[ChristianCiach]

Another workaround: Instead using the native OIDC support of argo-cd, we could use Dex as an intermediate.

For example, instead of configuring this...

oidc.config:
  name: Company-SSO
  issuer: https://company-keycloak/realms/some-realm
  clientID: argocd
  cliClientID: argocd-cli
  clientSecret: $oidc.keycloak.clientSecret
  requestedScopes: ["openid", "microprofile-jwt"]

We would instead configure this:

dex.config:
  connectors:
    - type: oidc
      id: oidc
      name: Company-SSO
      config:
        issuer: https://company-keycloak/realms/some-realm
        clientID: argocd
        clientSecret: $oidc.keycloak.clientSecret
        insecureEnableGroups: true
        getUserInfo: true
        scopes:
          - openid
          - microprofile-jwt

Don't forget to add /api/dex/callback as a valid redirect-uri to your oidc-client in keycloak.

This works because id-tokens issued by Dex are valid for 24h.

[DrPsychick]

I spent a few hours trying to understand the code and if it could be fixed on server-side, like suggested in some issue/PR/thread I cannot find anymore. The answer (as stated before) is, that it needs to be done in the UI and before that, some other PRs/fixes need to be merged (#8325 #15004). It is also mentioned in the code: https://github.com/argoproj/argo-cd/blob/8ca96a9d0e9f26821e72a163c5630a25f34ee1ce/util/session/sessionmanager.go#L543-L547

With that, currently the best workaround is through using Dex (which also, for example, supports GitLab SSO: https://dexidp.io/docs/connectors/gitlab/).

Thanks to all the people posting their configs here, it's quite simple to setup.

[bofeng96]

Change the Access Token lifespan to xx minutes. Waiting for long term safe solution

[Hronom]

Change the Access Token lifespan to xx minutes. Waiting for long term safe solution

It's not always possible to set in OIDC provider. So this not works in many cases.

We need to have support for refresh token. There no other way.

[bofeng96]

Change the Access Token lifespan to xx minutes. Waiting for long term safe solution

It's not always possible to set in OIDC provider. So this not works in many cases.

We need to have support for refresh token. There no other way.

I'm following the temporary solution provided in this thread and waiting for the optimal solution. What dissatisfaction do you have?

[rouke-broersma]

Change the Access Token lifespan to xx minutes. Waiting for long term safe solution

It's not always possible to set in OIDC provider. So this not works in many cases.

We need to have support for refresh token. There no other way.

I'm following the temporary solution provided in this thread and waiting for the optimal solution. What dissatisfaction do you have?

Extending access token lifetime is a big no no, it is not a valid workaround.

[michaelajr]

Bottom line is that argocd needs to support the refresh_token flow implicitly w/o the user knowing or seeing. Only after the refresh_token has fully expired should the user be redirected to login. This is a huge issue for us.

[andrii-korotkov-verkada]

Would this refresh token be universal for all kinds of integrations, or each thing like keycloak would needs its own implementation?

[Hronom]

@andrii-korotkov-verkada yes this is standard for OAuth 2.0 https://oauth.net/2/grant-types/refresh-token/

[andrii-korotkov-verkada]

Sounds good. I'll look into this more. Btw, why is default Keycloak token expiration is 5 minutes? That's so small. Shouldn't something like 12 hours be more reasonable?

[luispabon]

There's a whole load of security reasons around those defaults that are perhaps better suited for a discussion elsewhere other than this ticket.

[andrii-korotkov-verkada]

I see. Feel free to message me at CNCF Slack if you want to discuss further. I'm Andrii Korotkov (Verkada) there.

[rouke-broersma]

Sounds good. I'll look into this more. Btw, why is default Keycloak token expiration is 5 minutes? That's so small. Shouldn't something like 12 hours be more reasonable?

It's not just keycloak, I use Authentik which has the same default. Regardless of token expiration time, argocd should implement background token refresh. A short token lifetime just makes the issue more annoying.

See for example this recommendation by Google apigee https://cloud.google.com/apigee/docs/api-platform/antipatterns/oauth-long-expiration#:~:text=An%20access%20token%20should%20have,lifetime%20of%20the%20refresh%20token.

[andrii-korotkov-verkada]

I see. I'll bring this up in the contributors meeting and see if there are any objections or concerns with having an auto-refresh.

[Hronom]

@andrii-korotkov-verkada just for your info, this is actually considered as best security practices to have an access token to be short lived, sometimes it's even 1 minute. As @luispabon mention https://github.com/argoproj/argo-cd/issues/12189#issuecomment-2483539625 there a list of security reasons. Also I would say this is a hard security ask, something that may be not even adjustable/discussable with security team.

So refresh token considered to have more live time and usually app using refresh token to get short live access token.

If Argo CD will have standard OAuth 2.0 flow this will be solution not only for Keycloack, but for JumpCloud, Okta and others SSO.

Also(I'm not sure here), but seems like Argo Workflows already have support for it https://argo-workflows.readthedocs.io/en/latest/argo-server-sso/

[fe80]

Hello,

When we pass to dex server we have a limit about the jwt token (#14250). In my case If I move my Gitlab authentication to the dex server, that solve this issue but I can't use a token generate by a Gitlab CI for argocd cli.

I'm not an expert about the subjet and I'm very confuse about the double support, oidc.config and dex server, and limits about both of them.

The good solution is maybe help on #14250 and only support oidc with dex ?

[michaelajr]

The oidc.config and dex.config both need to support refresh tokens exclusively of each other. It is standard behavior. Many companies can not use dex because it goes agains their cyber policy that the internal enterprise IdP be the only IdP that issues tokens. ArgoCD must integrate with their internal IdP.