Eclipse Che: Unable to open User workspace: Giving Unauthorized error in Kube-rbac-proxy

[debkantap]

Describe the bug

Tried to install Eclipse Che in GKE with Keycloak as OIDC provider.

Able to successfully deploy eclipse che on GKE For GKE enabled External OIDC identity and integrated with keycloak Dashboard is opening and user workspace is successfully getting created While opening the user workspace, we are getting 'Unauthorized' error In the kube-rbac-proxy container of Che gateway we are getting following error: -------------------------error------------------------ Unable to authenticate the request due to an error: invalid bearer token

Open in Logs Explorer { insertId: "i5jui1z539rwxkh3" jsonPayload: {2} labels: {9} logName: "projects/dev-experience-395309/logs/stderr" receiveTimestamp: "2023-09-13T07:03:47.340391668Z" resource: { labels: { container_name: "kube-rbac-proxy" cluster_name: "cluster-7" location: "us-central1-b" pod_name: "che-gateway-8855cb995-pfrld" project_id: "dev-experience-395309" namespace_name: "eclipse-che" } type: "k8s_container"

---------------error------------------------

Che version

7.72

Steps to reproduce

Steps:

1. Installed Eclipse Che on GKE with following command:

chectl server:deploy --platform k8s --che-operator-cr-patch-yaml che-operator-cr-patch.yaml --domain 34.70.xxx.xx.nip.io

1. Eclipse Che was integrated with keycloak as OIDC provider It installed successfully and after successful login, while creating user workspace we are getting the error mentioned above. The che-operator-cr-patch.yaml file below------

kind: CheCluster
apiVersion: org.eclipse.che/v2
spec:
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_USERNAME__CLAIM: email
        serverExposureStrategy: 'multi-host'
        workspaceNamespaceDefault: 'cheuser'
        ingressStrategy: 'single-host'
        CHE_INFRA_KUBERNETES_MASTER__URL: https://gke-oidc-envoy.anthos-identity-service

    dashboard:
      deployment:
        containers:
          -  env:  
              - name: KUBERNETES_PORT
                value: "tcp://30.90.rt.rr:443"
              - name: KUBERNETES_PORT_443_TCP_ADDR
                value: "30.90.rt.rr"
              - name: KUBERNETES_PORT_443_TCP
                value: "tcp://30.90.rt.rr:443"
              - name: KUBERNETES_SERVICE_HOST
                value: "30.90.rt.rr"                

  networking:
    domain: 34.70.xxx.xx.nip.io
    annotations:
        kubernetes.io/ingress.class: nginx
        nginx.ingress.kubernetes.io/proxy-body-size: "100m"
        nginx.ingress.kubernetes.io/proxy-buffer-size: "256k"
        nginx.ingress.kubernetes.io/proxy-buffering: "on"
        nginx.ingress.kubernetes.io/proxy-buffers-number: "4"
        nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "1024m"
        nginx.ingress.kubernetes.io/ssl-redirect: "false"

    auth:
      externalIdentityProvider: true
      openShiftoAuth: false
      oAuthClientName: "kubenew-client-id"
      oAuthSecret: "Qz32dddddddRxuOW"
      identityProviderURL: "https://testagain.co.in/auth/realms/kubernetes-che-realm"

---

Expected behavior

Workspace must open gracefully and successfully

Runtime

other (please specify in additional context)

Screenshots

No response

Installation method

chectl/latest

Environment

GCE

Eclipse Che Logs

-------------------------error------------------------
Unable to authenticate the request due to an error: invalid bearer token

[Open in Logs Explorer](https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22devss-experience-395309%22%0Aresourcsse.labels.location%3D%22us-central1-b%22%0Aresource.labels.cluster_name%3D%22cluster-7%22%0Aresource.labels.namespace_name%3D%22eclipse-che%22%0Aresource.labels.pod_name:%22che-gateway-%22%20severity%3E%3DDEFAULT;timeRange=2023-09-13T07:03:44.577158311Z%2F2023-09-13T07:03:44.577158311Z--PT1H;pinnedLogId=2023-09-13T07:03:44.577158311Z%9992Fi5jui1ssz539rwxkh3?project=dev-experience-395309)
{
insertId: "i5jui1z539rwxkh3"
jsonPayload: {2}
labels: {9}
logName: "projects/dev-experience-395309/logs/stderr"
receiveTimestamp: "2023-09-13T07:03:47.340391668Z"
resource: {
labels: {
container_name: "kube-rbac-proxy"
cluster_name: "cluster-7"
location: "us-central1-b"
pod_name: "che-gateway-8855cb995-pfrld"
project_id: "dev-experience-395309"
namespace_name: "eclipse-che"
}
type: "k8s_container"

---------------error------------------------

Additional context

No response

[debkantap]

Can you please give some pointers. Tried multiple option, but not succeeded. Need help.

[l0rd]

@debkantap thanks for reporting this issue. @tolusha any clue?

[debkantap]

Thanks for responding ...Any light on this issue will be very helpful. Many Thanks!!

[debkantap]

Hello..Is this a bug? Can you please advise..we can't move further.

[serhii-kuzniechykov]

Hello We have that same behavior on our GKE with Che version 7.77 deployed in the same way.

[serhii-kuzniechykov]

@debkantap Hello. Do you solve this issue?

[debkantap]

No @serhii-kuzniechykov ..we have not further troubleshoot this as I thought this is a bug. We went ahead and deployed che on vanilla k8s. Please post if you have luck on this. Thanks

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.