HTTP Error code 500 (K3s with keycloak as oidc provider)

[pppaapp]

Describe the bug

If the web interface is accessed again after a period of time after logging in, a 500 error is output.

io.jsonwebtoken.ExpiredJwtException: JWT expired at 2024-01-16T03:29:09Z. Current time: 2024-01-16T03:58:40Z, a difference of 1768907 milliseconds.  Allowed clock skew: 3000 milliseconds.
    io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:448)
    io.jsonwebtoken.impl.DefaultJwtParser.parse(DefaultJwtParser.java:550)
    io.jsonwebtoken.impl.DefaultJwtParser.parseClaimsJws(DefaultJwtParser.java:610)
    io.jsonwebtoken.impl.ImmutableJwtParser.parseClaimsJws(ImmutableJwtParser.java:173)
    org.eclipse.che.multiuser.oidc.filter.OidcTokenInitializationFilter.processToken(OidcTokenInitializationFilter.java:80)
    org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:127)
    org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
    com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
    com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)

Che version

7.80@latest

Steps to reproduce

1. open the link of che dashboard
2. redirect to keycloak login page, enter username and password
3. redirect to che dashboard
4. 5 minutes later, che dashboard reports "HTTP Error code 500. Endpoint which throws an error https://example.com/api/kubernetes/namespace/provision. Check the browser logs message."

Expected behavior

.

Runtime

Kubernetes (vanilla)

Screenshots

No response

Installation method

chectl/latest

Environment

Linux

Eclipse Che Logs

No response

Additional context

No response

[pppaapp]

Deleting keycloak cookies resolves the issue

But I can't keep deleting cookies every time something goes wrong...

[AObuchow]

@olexii4 @akurinnoy @tolusha any ideas here? I'm not yet too familiar with JWT's or ODIC providers

[akurinnoy]

cc @ibuziuk @dkwon17

[monaka]

Looks similar to #22130 and #21376 ?

[monaka]

@pppaapp I'm getting a same issue.

Deleting keycloak cookies resolves the issue

In my case, deleting the cookie _oauth2_proxy that is given from my Che instance resolves the issue. (not Keycloak's)

Just guess: oauth2-proxy (in che-gateway) doesn't refresh cookie.

[monaka]

Referring to https://oauth2-proxy.github.io/oauth2-proxy/configuration/overview#command-line-options , --cookie-refresh is available for OIDC. And the default value is 0 which means disabled. I think this is the root cause of this issue.

Referring to code in che-operator, we have no way to set cookie-refresh. https://github.com/eclipse-che/che-operator/blob/1ae01423c15e14cb9aff32c65bc2c71686a99ca5/api/v2/checluster_types.go#L581C36-L581C55

[monaka]

A rough workaround for Keycloak admins: Increase Access Token Lifespan (and may SSO Session Idle also).

[monaka]

After some tries and errors, I got a little bit better workaround.

Set cookieExpireSeconds shorter than Access Token Lifespan.

For example: If "10 minutes" was set as your Access Token Lifespan in your realm. Set cookieExpireSeconds like follows.

kind: CheCluster
apiVersion: org.eclipse.che/v2
spec:
  networking:
    auth:
      gateway:
        oAuthProxy:
          cookieExpireSeconds: 300
          # means 5 minutes (shorter than 10 minutes that is defined as `Access Token Lifespan`)

[tolusha]

@monaka How to configure oauth2-proxy [1] [1] https://github.com/eclipse/che/issues/22777#issuecomment-1903918634

[monaka]

@tolusha Thank you for your link.

In this case, I think we can fix this (and related) issue by setting reasonable cookieExpreSeconds https://github.com/eclipse/che/issues/22765#issuecomment-1951482787 , without envvars.

I lived in my Che (7.82.0-next) instance with fixed cookieExpireSeconds yesterday. It didn't reproduce this issue.

Even though I don't see why issues are reported from uses that use Keycloak only, I suppose this issue can be closed as resolved.

[maheshrajrp]

Facing this same issue. @monaka I have a question, shouldn't this be resolved at client code, where Eclipse Che Dashboard Redirects and begins the SignIn work flow again ? Throwing an exception that doesn't convey any meaning to the user, or the possible actions they could try seems to be flawed ?