Dashboard not reachable after successfully deploying eclipse che on AKS cluster

[kuldeeparora89]

Describe the bug

The dashboard page is not reachable even after the successful installation of the che in AKS.

Che version

next (development version)

Steps to reproduce

chectl server:deploy --platform=k8s --installer=operator --che-operator-cr-patch-yaml=C:\Users\Act9\Desktop\che.yaml --skip-oidc-provider-check --domain=impaktapps.com --k8spodreadytimeout=600000

Expected behavior

A dashboard page should open up.

Runtime

other (please specify in additional context)

Screenshots

DNS Zone Entries :

Installation method

chectl/next

Environment

Azure

Eclipse Che Logs

PS C:\Users\Act9> chectl server:deploy --platform=k8s --installer=operator --che-operator-cr-patch-yaml=C:\Users\Act9\Desktop\che.yaml --skip-oidc-provider-check --domain=impaktapps.com --k8spodreadytimeout=600000
› Current Kubernetes context: 'eclipse-che-2'
  v Verify Kubernetes API...[1.23]
  v Kubernetes preflight checklist
    v Verify if kubectl is installed...[OK]
    v Verify domain is set...[OK]
  v Create Namespace eclipse-che...[Created]
  v Install Cert Manager v1.8.2
    v Apply resources...[Created]
    v Wait for Cert Manager pods ready...[OK]
  v Start following Eclipse Che installation logs...[OK]
  v Deploy Eclipse Che operator
    v Install Dev Workspace operator
      v Create Namespace devworkspace-controller...[Created]
      v Create Dev Workspace operator resources...[Created]
      v Wait for Dev Workspace operator ready...[OK]
      v Create DevWorkspaceOperatorConfig devworkspace-operator-config...[Created]
    v Create ServiceAccount che-operator...[Created]
    v Create RBAC
      v Create Role che-operator-leader-election...[Created]
      v Create Role che-operator...[Created]
      v Create RoleBinding che-operator-leader-election...[Created]
      v Create RoleBinding che-operator...[Created]
      v Create RoleBinding eclipse-che-che-operator...[Created]
      v Create RoleBinding eclipse-che-che-operator...[Created]
    v Wait for Cert Manager pods ready...[OK]
    v Create Certificate che-operator-serving-cert...[Created]
    v Create Issuer che-operator-selfsigned-issuer...[Created]
    v Create Service che-operator-service...[Created]
    v Create CRD checlusters.org.eclipse.che...[Created]
    v Waiting...[OK]
    v Create Deployment che-operator...[Created]
    v Eclipse Che Operator pod bootstrap
      v Scheduling...[OK]
      v Downloading images...[OK]
      v Starting...[OK]
    v Create ValidatingWebhookConfiguration org.eclipse.che...[Created]
    v Create MutatingWebhookConfiguration org.eclipse.che...[Created]
    v Create CheCluster Custom Resource...[Created]
  v Wait for Eclipse Che ready
    v Postgres pod bootstrap
      v Scheduling...[OK]
      v Downloading images...[OK]
      v Starting...[OK]
    v Devfile Registry pod bootstrap
      v Scheduling...[OK]
      v Downloading images...[OK]
      v Starting...[OK]
    v Plugin Registry pod bootstrap
      v Scheduling...[OK]
      v Downloading images...[OK]
      v Starting...[OK]
    v Dashboard pod bootstrap
      v Scheduling...[OK]
      v Downloading images...[OK]
      v Starting...[OK]
    v Gateway pod bootstrap
      v Scheduling...[OK]
      v Downloading images...[OK]
      v Starting...[OK]
    v Eclipse Che Server pod bootstrap
      v Scheduling...[OK]
      v Downloading images...[OK]
      v Starting...[OK]
    v Wait Eclipse Che active...[OK]
  v Eclipse Che status check...[OK]
  v Retrieving Eclipse Che self-signed CA certificate...[OK: C:\Users\Act9\AppData\Local\Temp\cheCA.crt]
  v Prepare post installation output...[OK]
  v Show important messages
    v Eclipse Che next has been successfully deployed.
    v Documentation             : https://www.eclipse.org/che/docs/
    v -------------------------------------------------------------------------------
    v Users Dashboard           : https://impaktapps.com/dashboard/
    v -------------------------------------------------------------------------------
    v Plug-in Registry          : https://impaktapps.com/plugin-registry/v3/
    v Devfile Registry          : https://impaktapps.com/devfile-registry/
    v -------------------------------------------------------------------------------
Command server:deploy has completed successfully in 09:14.

Additional context

Runtime : AKS Kubernetes version : 1.23.12

che.yaml

spec:
  networking:
    auth:
      identityProviderURL: https://sts.windows.net/c21d796f-015a-4b0f-9c5f-800f7f6237f9/v2.0/
      identityToken: access_token
      oAuthClientName: e189add7-57c5-4d0c-9073-34b5a60ffde0
      oAuthSecret: ccc6a45f-0a35-4d14-9a1f-e1400a51b8dd
      oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read
  components:
    cheServer:
      extraProperties:
        CHE_OIDC_AUTH__SERVER__URL: https://sts.windows.net/c21d796f-015a-4b0f-9c5f-800f7f6237f9/v2.0/
        CHE_OIDC_EMAIL__CLAIM: unique_name

[ibuziuk]

hello, could you please take a look at the following blog post which should have valid instructions for AKS deployment - https://che.eclipseprojects.io/2022/07/25/@karatkep-installing-eclipse-che-on-aks.html

[kuldeeparora89]

Hi @ibuziuk , I followed the instructions from this page only, hence was able to install the che successfully. As attached in the logs above all pods are also running fine. But still, the dashboard is not opening & it is unreachable. So I am not able to identify the root cause of the issue. My guess is that it might be due to some missing entries in DNS record set but I am not sure.

[kuldeeparora89]

I was able to proceed ahead by installing nginx ingress & mapping its external IP in DNS Zone. Also I have added below redirect URI in che app registration : "https://impaktapps.com/oauth/callback".

Now login page of Microsoft is opening up but after submit it is throwing below error :

Kindly suggest.

[karatkep]

@kuldeeparora89 could you please check and share oauth-proxy logs from che-gateway pod? It should show the root of the issue.

[kuldeeparora89]

@karatkep, oauth-proxy logs -

[2022/12/18 06:58:24] [provider.go:55] Performing OIDC Discovery...
[2022/12/18 06:58:24] [proxy.go:89] mapping path "/" => upstream "http://127.0.0.1:8081/"
[2022/12/18 06:58:24] [oauthproxy.go:162] OAuthProxy configured for OpenID Connect Client ID: e0e181e5-a696-45b5-920c-d2f90bcc9c46
[2022/12/18 06:58:24] [oauthproxy.go:168] Cookie settings: name:_oauth2_proxy secure(https):true httponly:false expiry:24h0m0s domains:.com path:/ samesite: refresh:disabled
[2022/12/18 06:58:24] [oauthproxy.go:476] Skipping auth - Method:  | Path: ^/plugin-registry|^/devfile-registry|^/$|/healthz$|^/dashboard/static/preload
[2022/12/18 07:13:35] [oauthproxy.go:959] No valid authentication in request. Initiating login.
10.244.1.10:46508 - 62cbb25f53e36dc4878a091da36deede - - [2022/12/18 07:13:35] impaktapps.com GET - "/dashboard/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.54" 302 420 0.001
[2022/12/18 07:13:43] [oauthproxy.go:816] &{GET /oauth/callback?code=0.AUoAb3kdwloBD0ucX4APf2I3-eWB4eCWprVFkgzS-QvMnEZKABc.AgABAAIAAAD--DLA3VO7QrddgJg7WevrAgDs_wQA9P-nernpO243VgdRDNWdxkWDL-AZ5HxxUkFCNd9Dowjb-M05snghDCBaUkfxrbsrrfs5_RRVt3htrLHue3E8UIsC7vLCXWWwjzhRVtZeZleSM5r82MQqttYicFMje5xi7vni22pgVabYY2e7JakrGeWaFppDhPlVM8TzjZCsAwKbirnX-YwR6ncnJlKOSa5EJgYhxccHW6twjXSiBwKS-90QgNWCifEOSGw5YpqsXCbQ1_Bk_pWXMiNzMdMsQp77L-YacCFerr9ymd672-eFVHqhKwJdMacMpUYBscaBrpvT8EV0_v247KQaHuBfV-oCQMZMdEMaXYesNE9ja8D2BYv5v_-RoBPq7xnAf_HiFqVqfa9hIzAvrPT7qWxLVUXu5y5KjCNwMUT32vCuL6CnjjnWx9hWLJLahSRMTaaC9OBYz0dNEqjKDSzu1ORo9J0mzmjJ8tH91aX5xbHkC0twbfcf9nWgdHkpnz0kP0po3vmg7zUlY9Ly8MYHci4tgrTzUECBXod05dNJXqXLjMsUPnwBcYcD-oi851h4tVS4CJI1nDhrTsqVdJHPDccVkUyu9nNXIWdydu_q_hp4CE_7zY6wCJMJHZhayQ5rZbWFGenv4lq4xH2q0tFhqVZcEcc7_KV0XqaRWAH9XOjvS1oNh6ZMuoIvOAI0hGi7IWZCu1OtWRt8pV2w2amKX_h-DthT98H7KKIEmOGGKvvtEzQa6tZTZdzfCMgGfhcWdXyAIqqNKzHJCGnEu8op5SsLpSamzggiwZxXyiJEmn-ITZYkzm2kwgO1kLk6hv0PvMAi0jHG-MlVgQuXAit_H61N_LzJpNW72MDwrLLBUSSX5kwmsOTOxUwFfR-BaF6dFFN7SEkYokO5p9y_Ga_Nhokmv9GYp3YDm53S6EY0MRXP9jtRd_NHJKRP0unfxdzNe8gjzvIIwExpO0PakMcZIs3YWyHOsynE9Fcy-7VhuOiXYFanPdHGs9K5G3iXsLUkjs8&state=6R-m9jQBuX6HPIuoRXazLHT8FpMCzlb0mlj7JFIZC0I%3a%2fdashboard%2f&session_state=cf3c4f5f-25bc-4849-99e5-355596c9b0a9 HTTP/1.1 1 1 map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[en-US,en;q=0.9] Referer:[https://login.microsoftonline.com/] Sec-Ch-Ua:["Not?A_Brand";v="8", "Chromium";v="108", "Microsoft Edge";v="108"] Sec-Ch-Ua-Mobile:[?0] Sec-Ch-Ua-Platform:["Windows"] Sec-Fetch-Dest:[document] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[cross-site] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.54] X-Forwarded-For:[10.224.0.6] X-Forwarded-Host:[impaktapps.com] X-Forwarded-Port:[443] X-Forwarded-Proto:[https] X-Forwarded-Scheme:[https] X-Real-Ip:[10.224.0.6] X-Request-Id:[a97f1326570241fda67ad6ba83d4b773] X-Scheme:[https]] {} <nil> 0 [] false impaktapps.com map[code:[0.AUoAb3kdwloBD0ucX4APf2I3-eWB4eCWprVFkgzS-QvMnEZKABc.AgABAAIAAAD--DLA3VO7QrddgJg7WevrAgDs_wQA9P-nernpO243VgdRDNWdxkWDL-AZ5HxxUkFCNd9Dowjb-M05snghDCBaUkfxrbsrrfs5_RRVt3htrLHue3E8UIsC7vLCXWWwjzhRVtZeZleSM5r82MQqttYicFMje5xi7vni22pgVabYY2e7JakrGeWaFppDhPlVM8TzjZCsAwKbirnX-YwR6ncnJlKOSa5EJgYhxccHW6twjXSiBwKS-90QgNWCifEOSGw5YpqsXCbQ1_Bk_pWXMiNzMdMsQp77L-YacCFerr9ymd672-eFVHqhKwJdMacMpUYBscaBrpvT8EV0_v247KQaHuBfV-oCQMZMdEMaXYesNE9ja8D2BYv5v_-RoBPq7xnAf_HiFqVqfa9hIzAvrPT7qWxLVUXu5y5KjCNwMUT32vCuL6CnjjnWx9hWLJLahSRMTaaC9OBYz0dNEqjKDSzu1ORo9J0mzmjJ8tH91aX5xbHkC0twbfcf9nWgdHkpnz0kP0po3vmg7zUlY9Ly8MYHci4tgrTzUECBXod05dNJXqXLjMsUPnwBcYcD-oi851h4tVS4CJI1nDhrTsqVdJHPDccVkUyu9nNXIWdydu_q_hp4CE_7zY6wCJMJHZhayQ5rZbWFGenv4lq4xH2q0tFhqVZcEcc7_KV0XqaRWAH9XOjvS1oNh6ZMuoIvOAI0hGi7IWZCu1OtWRt8pV2w2amKX_h-DthT98H7KKIEmOGGKvvtEzQa6tZTZdzfCMgGfhcWdXyAIqqNKzHJCGnEu8op5SsLpSamzggiwZxXyiJEmn-ITZYkzm2kwgO1kLk6hv0PvMAi0jHG-MlVgQuXAit_H61N_LzJpNW72MDwrLLBUSSX5kwmsOTOxUwFfR-BaF6dFFN7SEkYokO5p9y_Ga_Nhokmv9GYp3YDm53S6EY0MRXP9jtRd_NHJKRP0unfxdzNe8gjzvIIwExpO0PakMcZIs3YWyHOsynE9Fcy-7VhuOiXYFanPdHGs9K5G3iXsLUkjs8] session_state:[cf3c4f5f-25bc-4849-99e5-355596c9b0a9] state:[6R-m9jQBuX6HPIuoRXazLHT8FpMCzlb0mlj7JFIZC0I:/dashboard/]] map[] <nil> map[] 10.244.1.10:46508 /oauth/callback?code=0.AUoAb3kdwloBD0ucX4APf2I3-eWB4eCWprVFkgzS-QvMnEZKABc.AgABAAIAAAD--DLA3VO7QrddgJg7WevrAgDs_wQA9P-nernpO243VgdRDNWdxkWDL-AZ5HxxUkFCNd9Dowjb-M05snghDCBaUkfxrbsrrfs5_RRVt3htrLHue3E8UIsC7vLCXWWwjzhRVtZeZleSM5r82MQqttYicFMje5xi7vni22pgVabYY2e7JakrGeWaFppDhPlVM8TzjZCsAwKbirnX-YwR6ncnJlKOSa5EJgYhxccHW6twjXSiBwKS-90QgNWCifEOSGw5YpqsXCbQ1_Bk_pWXMiNzMdMsQp77L-YacCFerr9ymd672-eFVHqhKwJdMacMpUYBscaBrpvT8EV0_v247KQaHuBfV-oCQMZMdEMaXYesNE9ja8D2BYv5v_-RoBPq7xnAf_HiFqVqfa9hIzAvrPT7qWxLVUXu5y5KjCNwMUT32vCuL6CnjjnWx9hWLJLahSRMTaaC9OBYz0dNEqjKDSzu1ORo9J0mzmjJ8tH91aX5xbHkC0twbfcf9nWgdHkpnz0kP0po3vmg7zUlY9Ly8MYHci4tgrTzUECBXod05dNJXqXLjMsUPnwBcYcD-oi851h4tVS4CJI1nDhrTsqVdJHPDccVkUyu9nNXIWdydu_q_hp4CE_7zY6wCJMJHZhayQ5rZbWFGenv4lq4xH2q0tFhqVZcEcc7_KV0XqaRWAH9XOjvS1oNh6ZMuoIvOAI0hGi7IWZCu1OtWRt8pV2w2amKX_h-DthT98H7KKIEmOGGKvvtEzQa6tZTZdzfCMgGfhcWdXyAIqqNKzHJCGnEu8op5SsLpSamzggiwZxXyiJEmn-ITZYkzm2kwgO1kLk6hv0PvMAi0jHG-MlVgQuXAit_H61N_LzJpNW72MDwrLLBUSSX5kwmsOTOxUwFfR-BaF6dFFN7SEkYokO5p9y_Ga_Nhokmv9GYp3YDm53S6EY0MRXP9jtRd_NHJKRP0unfxdzNe8gjzvIIwExpO0PakMcZIs3YWyHOsynE9Fcy-7VhuOiXYFanPdHGs9K5G3iXsLUkjs8&state=6R-m9jQBuX6HPIuoRXazLHT8FpMCzlb0mlj7JFIZC0I%3a%2fdashboard%2f&session_state=cf3c4f5f-25bc-4849-99e5-355596c9b0a9 <nil> <nil> <nil> 0xc000396810} AuthFailure Invalid authentication via OAuth2: unable to obtain CSRF cookie

[karatkep]

@kuldeeparora89, perhaps the issue with domain whitelisting. Could you please share configuration for oauth-proxy. It should be in ConfigMaps

[kuldeeparora89]

@karatkep ,

oauth-proxy.cfg-

proxy_prefix = "/oauth"
http_address = ":8080"
https_address = ""
provider = "oidc"
redirect_url = "https://impaktapps.com/oauth/callback"
oidc_issuer_url = "https://sts.windows.net/c21d796f-015a-4b0f-9c5f-800f7f6237f9/v2.0/"
insecure_oidc_skip_issuer_verification = true
ssl_insecure_skip_verify = true
upstreams = [
    "http://127.0.0.1:8081/"
]
client_id = "e0e181e5-a696-45b5-920c-d2f90bcc9c46"
client_secret = "5459c965-c8e4-4958-849e-47baafe61909"
cookie_secret = "TFFXcm92VFVGbnJ5TmNRbw=="
cookie_expire = "24h0m0s"
email_domains = "*"
cookie_httponly = false
skip_provider_button = true
whitelist_domains = ".com"
cookie_domains = ".com"
skip_auth_routes = "^/plugin-registry|^/devfile-registry|^/$|/healthz$|^/dashboard/static/preload"
pass_access_token = true
scope = "openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read"

yaml-

kind: ConfigMap
apiVersion: v1
metadata:
  name: che-gateway-config-oauth-proxy
  namespace: eclipse-che
  uid: 31d4601d-796f-4686-8e8c-4a437306eee1
  resourceVersion: '150309'
  creationTimestamp: '2022-12-18T06:57:59Z'
  labels:
    app.kubernetes.io/component: che-gateway
    app.kubernetes.io/instance: che
    app.kubernetes.io/managed-by: che-operator
    app.kubernetes.io/name: che
    app.kubernetes.io/part-of: che.eclipse.org
  ownerReferences:
    - apiVersion: org.eclipse.che/v2
      kind: CheCluster
      name: eclipse-che
      uid: d2391f76-a746-44c8-ac43-324df211984f
      controller: true
      blockOwnerDeletion: true
  managedFields:
    - manager: manager
      operation: Update
      apiVersion: v1
      time: '2022-12-18T06:57:59Z'
      fieldsType: FieldsV1
      fieldsV1:
        f:data:
          .: {}
          f:oauth-proxy.cfg: {}
        f:metadata:
          f:labels:
            .: {}
            f:app.kubernetes.io/component: {}
            f:app.kubernetes.io/instance: {}
            f:app.kubernetes.io/managed-by: {}
            f:app.kubernetes.io/name: {}
            f:app.kubernetes.io/part-of: {}
          f:ownerReferences:
            .: {}
            k:{"uid":"d2391f76-a746-44c8-ac43-324df211984f"}: {}
data:
  oauth-proxy.cfg: "\nproxy_prefix = \"/oauth\"\nhttp_address = \":8080\"\nhttps_address = \"\"\nprovider = \"oidc\"\nredirect_url = \"https://impaktapps.com/oauth/callback\"\noidc_issuer_url = \"https://sts.windows.net/c21d796f-015a-4b0f-9c5f-800f7f6237f9/v2.0/\"\ninsecure_oidc_skip_issuer_verification = true\nssl_insecure_skip_verify = true\nupstreams = [\n\t\"http://127.0.0.1:8081/\"\n]\nclient_id = \"e0e181e5-a696-45b5-920c-d2f90bcc9c46\"\nclient_secret = \"5459c965-c8e4-4958-849e-47baafe61909\"\ncookie_secret = \"TFFXcm92VFVGbnJ5TmNRbw==\"\ncookie_expire = \"24h0m0s\"\nemail_domains = \"*\"\ncookie_httponly = false\nskip_provider_button = true\nwhitelist_domains = \".com\"\ncookie_domains = \".com\"\nskip_auth_routes = \"^/plugin-registry|^/devfile-registry|^/$|/healthz$|^/dashboard/static/preload\"\npass_access_token = true\nscope = \"openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read\"\n"

[karatkep]

I think the issue with

whitelist_domains = ".com"
cookie_domains = ".com"

@kuldeeparora89 , could you please try to re-work and re-deploy che to use another domain? For example, che.impaktapps.com? So, your chectl server:deploy command will contain --domain=che.impaktapps.com

[kuldeeparora89]

@karatkep , That fixed the token issue. But now facing 500 error.

Logs :

[2022/12/18 12:45:40] [oauthproxy.go:823] Error redeeming code during OAuth2 callback: could not get claim "groups": failed to fetch claims from profile URL: error making request to profile URL: unexpected status "401": {"error":{"code":"InvalidAuthenticationToken","message":"Access token validation failure. Invalid audience.","innerError":{"date":"2022-12-18T12:45:40","request-id":"53829317-1e1d-416c-a393-45aec165fb2d","client-request-id":"53829317-1e1d-416c-a393-45aec165fb2d"}}}
10.244.2.2:34570 - b23a11db49c6a74323526db2c6a90b32 - - [2022/12/18 12:45:40] che.impaktapps.com GET - "/oauth/callback?code=0.AUkAb3kdwloBD0ucX4APf2I3-eWB4eCWprVFkgzS-QvMnEZJAAA.AgABAAIAAAD--DLA3VO7QrddgJg7WevrAgDs_wQA9P-mq8x8KZMKmBZ4XveO2uNfo_tYht4WrJUmweq4EAATvbUoCDstvFCquZSfxL1PQgDBkgYjsEYoHBnb3_j7Vt_Z2FMTwJBOhRTptQBXWsxoZdEkmcaXQ7IklNKd6raOFS9kxmxg_jhhjN71-Ygm_XTKV0IdcDOggOaY0tkUJMmFd4zD3mCya4_XEMq4LZ5P9tDV5K9anDoTOIQIkhPjqxCYPA--J0tOtrTbx44q-UPByxuI0c8gNoh3nUIdzygqf_-NQN4Kfob-48pg9o_WizQQgjfiDMn_TFNrJunCmSJUGuQhxyocDWgP-0BJvJlXvwoNeJ3Bq10XqCOWuKAtivGKRWJhMT37VKBytcUSpfrWyzwlzCw1q74OfTK6uVSd-kEfhdHD_JpSHbzxnul4q2REELXSA4mMD53zU2klij5Crl_ZxCOMnCANtweBbu4eFE9hqDCGBw3eyIK5l2mn8YdJT3RixepyR71g15M0IBslUP2EMTb8coqEeDElUrF1FgWaXx9lTA46bIFWVxI4gJG6XeZ9RuobbtxapWHzbeJ0ZhAyX4S6bv1B177VV9HFnzk3goQzAEeElDghPYlF65Di2nG3VqmXdNeyqUZeaOJDukZ2fQ0gf0MvrT1nmXjogD_LDjLd-SywnrvOYAoUYzhAYUJKPC4o60I6lmchdmbgZHLXoluF-xFBbobUnA6UP0gpGiWkAgEH4PY62kcraz6wd-I1RH6T5ULPFvJfAjnGgjKAh0cIIxfPoelTc_v8JZ93s73TCVJ5w41hXPet3LmMbMLmg3CkzaLj0nyyzR_4Sn-RfbgZ5fEnttek8aTLMFuRcUhp6GXNThBcSZ0j3e9Xq0dZ2pXkUwQrYw&state=x5hv4OYfWTilDXVQvLjuZwKeO96XQ2FKhLCAKtSHMXI%3a%2fdashboard%2f&session_state=fb5d1862-c6bc-4d79-8620-88bedb358111" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.54" 500 2836 0.353

[karatkep]

@kuldeeparora89 the next step is to double check oauth-proxy.cfg and your App registration created in Azure Portal. You need to check following sections:

• Authentication: it should contain a proper redirect, for example:
• Certificates & secrets: it should contain a proper secret, for example:
• API permissions: it should contain the proper permissions, for example:

[kuldeeparora89]

@karatkep , All these are properly setup.

[karatkep]

@kuldeeparora89 could you please share oauth-proxy.cfg one more time?

[kuldeeparora89]

@karatkep , oauth-proxy.cfg :

proxy_prefix = "/oauth"
http_address = ":8080"
https_address = ""
provider = "oidc"
redirect_url = "https://che.impaktapps.com/oauth/callback"
oidc_issuer_url = "https://sts.windows.net/c21d796f-015a-4b0f-9c5f-800f7f6237f9/v2.0/"
insecure_oidc_skip_issuer_verification = true
ssl_insecure_skip_verify = true
upstreams = [
    "http://127.0.0.1:8081/"
]
client_id = "e0e181e5-a696-45b5-920c-d2f90bcc9c46"
client_secret = "a9W8Q~2EOOr67kVZcSHZryLqewgC1VYVp4R3ldd5"
cookie_secret = "UnBRODgzTUo4YzJkR0xSYw=="
cookie_expire = "24h0m0s"
email_domains = "*"
cookie_httponly = false
skip_provider_button = true
whitelist_domains = ".impaktapps.com"
cookie_domains = ".impaktapps.com"
skip_auth_routes = "^/plugin-registry|^/devfile-registry|^/$|/healthz$|^/dashboard/static/preload"
pass_access_token = true
scope = "openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read"

App registration :

[kuldeeparora89]

@karatkep , added Group.ReadAll permission as well. But the same issue :

[karatkep]

@kuldeeparora89 both oauth-proxy.cfg and App registration look correct to me.

Let's check a cookies size. Perhaps cookies is too big and Nginx cuts it off.

[kuldeeparora89]

@karatkep ,

Checked the cookie size :

Also checked nginx logs & could find any message related to cookie too large.

[kuldeeparora89]

@karatkep ,

Can it be an issue with OIDC URL ? Should we try below for identity provider & auth server url : https://login.microsoftonline.com/{tenant-id}/v2.0

[karatkep]

@kuldeeparora89, sure please try

[karatkep]

@kuldeeparora89 btw, what is the version of oauth-proxy?

[kuldeeparora89]

@karatkep , 7.4.0 . Image - quay.io/oauth2-proxy/oauth2-proxy:v7.4.0

Tried with below URL as well but issue is same - https://login.microsoftonline.com/{tenant-id}/v2.0

Created new app & kube cluster :

oauth-proxy.cfg

proxy_prefix = "/oauth"
http_address = ":8080"
https_address = ""
provider = "oidc"
redirect_url = "https://che.impaktapps.com/oauth/callback"
oidc_issuer_url = "https://login.microsoftonline.com/c21d796f-015a-4b0f-9c5f-800f7f6237f9/v2.0"
insecure_oidc_skip_issuer_verification = true
ssl_insecure_skip_verify = true
upstreams = [
    "http://127.0.0.1:8081/"
]
client_id = "7d576833-07f0-4011-8523-80703c111dc6"
client_secret = "Ofb8Q~llHvexwS9g9zUyZsKMcM4_0e.HbgGu0cix"
cookie_secret = "Y0YwMjRvZWh3all4R2F3VQ=="
cookie_expire = "24h0m0s"
email_domains = "*"
cookie_httponly = false
skip_provider_button = true
whitelist_domains = ".impaktapps.com"
cookie_domains = ".impaktapps.com"
skip_auth_routes = "^/plugin-registry|^/devfile-registry|^/$|/healthz$|^/dashboard/static/preload"
pass_access_token = true
scope = "openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read"

[karatkep]

@kuldeeparora89 I never tried 7.4.0 before. Could you please try 7.2.0? You need to scale down che-operator to 0. Then Update che-gateway deployment to change quay.io/oauth2-proxy/oauth2-proxy:v7.4.0 -> quay.io/oauth2-proxy/oauth2-proxy:v7.2.0

[karatkep]

@kuldeeparora89 ok, I just realised significant changes in ouath2-proxy v7.4.0 configuration for azure https://github.com/oauth2-proxy/oauth2-proxy/releases/tag/v7.4.0. As a short term solution, please use v7.2.0. I will provide a long term solution soon.

[kuldeeparora89]

@karatkep ,

I ran below command to change oauth-proxy to 7.2.0 -

1.kubectl scale deployment che-operator -n eclipse-che -replicas 0
2.kubectl patch checluster/eclipse-che --patch '{"spec": {"networking": {"auth": {"gateway": {"deployment": {"containers": [{"name": "oauth-proxy", "image": "quay.io/oauth2-proxy/oauth2-proxy:v7.2.0"}]}}}}}}' --type=merge -n eclipse-che
3.kubectl scale deployment che-operator -n eclipse-che -replicas 1

Now I am getting 404 error - proxy logs -

[2022/12/19 12:26:02] [oauthproxy.go:162] OAuthProxy configured for OpenID Connect Client ID: e0e181e5-a696-45b5-920c-d2f90bcc9c46
[2022/12/19 12:26:02] [oauthproxy.go:168] Cookie settings: name:_oauth2_proxy secure(https):true httponly:false expiry:24h0m0s domains:.impaktapps.com path:/ samesite: refresh:disabled
[2022/12/19 12:26:02] [oauthproxy.go:476] Skipping auth - Method:  | Path: ^/plugin-registry|^/devfile-registry|^/$|/healthz$|^/dashboard/static/preload
10.244.1.12:43736 - 899ae8a78332f73183d3dc3c834cc1d6 - Pankaj.Gupta@act21io.onmicrosoft.com [2022/12/19 12:30:13] che.impaktapps.com GET / "/dashboard" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.54" 404 19 0.001

Below command is also not working -

kubectl patch checluster/eclipse-che --patch '{"spec": {"networking": {"auth": {"gateway": {"deployment": {"containers": [{"name": "gateway", "image": "oauth2-proxy:v7.2.0"}]}}}}}}' --type=merge -n eclipse-che

[tolusha]

Container name must be oauth-proxy:

kubectl patch checluster/eclipse-che --patch '{"spec": {"networking": {"auth": {"gateway": {"deployment": {"containers": [{"name": "oauth-proxy", "image": "oauth2-proxy:v7.2.0"}]}}}}}}' --type=merge -n eclipse-che

[kuldeeparora89]

@tolusha , updated my original comment with correct command kubectl patch checluster/eclipse-che --patch '{"spec": {"networking": {"auth": {"gateway": {"deployment": {"containers": [{"name": "oauth-proxy", "image": "quay.io/oauth2-proxy/oauth2-proxy:v7.2.0"}]}}}}}}' --type=merge -n eclipse-che

[kuldeeparora89]

@karatkep @tolusha , Does the 404 error is due to Nginx configuration ?

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: ingress-nginx
    meta.helm.sh/release-namespace: eclipse-che
  creationTimestamp: "2022-12-19T15:46:27Z"
  finalizers:
  - service.kubernetes.io/load-balancer-cleanup
  labels:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    app.kubernetes.io/version: 1.5.1
    helm.sh/chart: ingress-nginx-4.4.0
  name: ingress-nginx-controller
  namespace: eclipse-che
  resourceVersion: "2419"
  uid: 0fa08a11-b3b8-4cce-88a8-287468fba2e1
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.0.29.211
  clusterIPs:
  - 10.0.29.211
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - appProtocol: http
    name: http
    nodePort: 30212
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    nodePort: 31170
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/name: ingress-nginx
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 20.204.251.205

[karatkep]

@kuldeeparora89 I don't think so. Request flow is nginx -> oauth-proxy -> traefik -> dashboard You got 404 in oauth-proxy logs.

[kuldeeparora89]

Thanks, @karatkep @tolusha for the support. Finally, dashboard opened up. I had to hit swagger API first "dashboard/api/swagger" and then afterward dashboard page also opened up. @karatkep , will wait for your response on how to configure oauth-proxy v7.4.0 with che AKS setup. The latest release of che is using 7.4.0 so it becomes very critical.

[karatkep]

Hi @kuldeeparora89, Sorry for delay, now I have time to continue... Could you please back the latest version of oauth-proxy and post here the actual error log?

[kuldeeparora89]

Hi @karatkep , Thanks for the support. But unfortunately, now I am not using eclipse CHE. As I found it very heavy for my use case & also specifically when Eclipse CHE switched the default editor from Theia to vs code. But I will post the logs once I get a chance to look at it again.

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[rohittamra]

Hi

I am facing the same issue. Can someone please help me?

Thanks Regards Rohit

[karatkep]

Hi

I am facing the same issue. Can someone please help me?

Thanks Regards Rohit

Hello @rohittamra ,

My recommendation is to open new issue and provide precise description and reproduction steps, log and console traces.

[netomi]

@kuldeeparora89 in your snippets you included some secrets, I would advise you to double-check the content your are posting and reverting still in use secrets.