search

eclipse-che / che

Kubernetes based Cloud Development Environments for Enterprise Teams
http://eclipse.org/che
Eclipse Public License 2.0
6.99k stars 1.19k forks source link

After installing EclipseChe on the Azure Cloud we get 500 Internal Server Error #22849

Closed KTzerras closed 8 months ago

KTzerras commented 8 months ago

Describe the bug

We installed EclipseChe on the Azure Cloud according to the instructions under

https://eclipse.dev/che/docs/stable/administration-guide/installing-che-on-microsoft-azure/

Hereby we have used software-machines.online as DOMAIN_NAME (as instructed in section "Configuring DNS on Microsoft Azure").

According to section "Installing Che on Microsoft Azure Kubernetes Service" we deployed EclipseChe by the command

chectl server:deploy --platform=k8s --che-operator-cr-patch-yaml=che-cluster-patch.yaml --skip-oidc-provider-check --skip-cert-manager --domain=ide.software-machines.online

whereas we used the file che-cluster-patch.yaml with the following content:

spec: networking: auth: identityProviderURL: "https://sts.windows.net/1e6eb10c-ccbd-416a-9410-XXXXXXXXXXX/v2.0/" identityToken: access_token oAuthClientName: 58affa1d-fdb7-4365-b9c1-XXXXXXXXXXXX
oAuthSecret: OZe8Q~BXRlJ3NQQi1TIWw4.XXXXXXXXXXXXXXXXX oAuthScope: openid email profile 6dae42f8-4368-4678-94ff-3960e28e3630/user.read gateway: deployment: containers:

All other instructions of page https://eclipse.dev/che/docs/stable/administration-guide/installing-che-on-microsoft-azure/ were executed 1:1.

Finally we navigated to the Che cluster instance via the command

chectl dashboard:open

and received 500 Internal Server Error in the Browser and in the log file we receive the following errors:

che-gateway_logs

Can you please advise? Thank you in advance

Che version

7.81@latest

Steps to reproduce

Enter the URL ide.software-machines.online in a Browser. 500 Internal Server Error appears.

Expected behavior

The EclipseChe IDE starts without error.

Runtime

other (please specify in additional context)

Screenshots

No response

Installation method

chectl/latest

Environment

Windows

Eclipse Che Logs

[2024/02/27 09:51:31] [oauthproxy.go:476] Skipping auth - Method:  | Path: ^/plugin-registry|^/devfile-registry|^/$|/healthz$|^/dashboard/static/preload|^/dashboard/assets/branding/loader.svg$
[2024/02/27 10:09:14] [oauthproxy.go:959] No valid authentication in request. Initiating login.
10.244.5.11:47898 - 6dfc75f49b8ac39c08bc5108742960aa - - [2024/02/27 10:09:14] ide.software-machines.online GET - "/dashboard/" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 302 434 0.000
[2024/02/27 10:09:16] [oauthproxy.go:823] Error redeeming code during OAuth2 callback: could not get claim "email": failed to fetch claims from profile URL: error making request to profile URL: unexpected status "401": {"error":{"code":"InvalidAuthenticationToken","message":"Access token validation failure. Invalid audience.","innerError":{"date":"2024-02-27T10:09:16","request-id":"4e688114-c4f3-4e84-86af-f81f75bffb81","client-request-id":"4e688114-c4f3-4e84-86af-f81f75bffb81"}}}
10.244.5.11:47898 - c050e69ab43b3195812c16ff3245f561 - - [2024/02/27 10:09:15] ide.software-machines.online GET - "/oauth/callback?code=0.AXkADLFuHr3MakGUEMNmXXrAex36r1i3_WVDucGrlF6q5g0MAeo.AgABAAIAAADnfolhJpSnRYB1SVj-Hgd8AgDs_wUA9P-R56HQDZOUaX_XdxOnw5KbUE31V7b7BWAIJVpjcsDIaDq5Iz6lfcziO_FjLB4n89ssE2aYfq1wcQxOsPGT1dkyeQqLtk24pk_xx8CwYOynZGOuqjM0-2RUthKWaJySG1rjkm94R8Y3yvWw8vDBFq2dmt_X9ZPFU1kRWrFPHIYPzkEC4mKbhFmME1GrCUDgQAbBzL1vQ2TP8KpymHs-4P8cmUSX7agVlLzQof1slDK6wCfNmPoml7RUULyEwjLNhmgMdAv2jqhCDw4b23dIh4SSiGmIRCMkWNq_RFNGJrnnkOTvBdpIaCZFg3a0Ry5p6GCWHkN-0h-kVrN3i94zMq5BVpWnST7FNd8_YJ6GAdGeOXyW7X4FjECLSoawwO75z2yO5v_ynyfIRN0KqhTcL9V_kqU_LSUnFrJ3RGBCcYQvJ4nNB_hl8kBQ5klzKaX_RhSBmKM67fYaeapUI8mZqY_bAbAKHc8-c0GoeJgmJYo5kC3uHAUassjNLX9RvKHYMoil4fxaNtw7cwe8RGLxXUUpgg4aeQqGDQ3WLwqKbgDzIwwrMb1Y_qAIZteyvKDXw_ht8CbG7saNJSaaL7i6udjq8_rBrX0JReVTpSzL2srxlI_DPoNRDOut-xxKt-WOkRAePiE7Ioe_hidTGpOa1EOCe8NQ96WUbsBqMtE5bbZofXNjCRuCj0hotVlbdCU-0IWiCFP6XdmoPYc8r33YI3AcHDQa6KOxQDHvlt75_uEQ8N4BSAucniyByd8ksN0lQftDDfGf5iJWnvNHdhMJ8c09YLKe58XZ&state=rrRr1B_bhzb6bxLc7ujhAQD-6N5F9huA-GrHWdvkaWE%3a%2fdashboard%2f&session_state=4c9c886f-d322-453e-9719-26a4d5196ee7" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 500 2836 0.393
[2024/02/27 10:09:16] [oauthproxy.go:959] No valid authentication in request. Initiating login.
10.244.5.11:47898 - ce03c59425078bde69afd17ad6216d0e - - [2024/02/27 10:09:16] ide.software-machines.online GET - "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 302 433 0.000
[2024/02/27 10:09:17] [oauthproxy.go:823] Error redeeming code during OAuth2 callback: could not get claim "email": failed to fetch claims from profile URL: error making request to profile URL: unexpected status "401": {"error":{"code":"InvalidAuthenticationToken","message":"Access token validation failure. Invalid audience.","innerError":{"date":"2024-02-27T10:09:17","request-id":"2a52f4ea-be82-4d94-aa1c-bc9c3fe041d9","client-request-id":"2a52f4ea-be82-4d94-aa1c-bc9c3fe041d9"}}}
10.244.5.11:47898 - e4c04af67152e69d0d43b22dd6fa8021 - - [2024/02/27 10:09:17] ide.software-machines.online GET - "/oauth/callback?code=0.AXkADLFuHr3MakGUEMNmXXrAex36r1i3_WVDucGrlF6q5g0MAeo.AgABAAIAAADnfolhJpSnRYB1SVj-Hgd8AgDs_wUA9P-2XNa9YZLhyAe5fI2sq6f8Uxutzy6gG0ZwKzUM0jy_hP3hIwk-EBZ_NmNSN5COsFFW9XPN7iQ_jdZiLhYXLX5UeKydkU-6iVqpE6IBiSCW5z5ftsc_sLTilkaVrqAeylzyWJveNgFiHqLY9-8Q5TwyJOxuFTjL3MVwMoEVkvKBx5qUtLtC3XHSRLNmJvvzJwRyuigvgTDo_nrjJ8EB6V_tdQtVVnVz1nIbiHOQKhAgDs66jp-HvG8AG4ViP46H05m-KrbU55PRRbmP-vdtzVNjA8ycdP5A0R_Z0yLRhRDLZXQj-SP4owAzQ2JNk_RFxldsyARDcp6EqlPXymjSN1HF76m7xNpRVTgER64jCd4guLRmYClYAgGcqYGjlnG171Z6z_h_hZmlqUAkaVh1svfwPRXY1hJPlt3E6N5cbQ2WSsC6HLzwjS_IUe4JLn5Ka3t5ag_ISoaSc-UWyJ_DFRwlf3ssXwnnhYThjOhgifwEvHdadHREhLOSCN7E4Fw_yLzWP4Bkn8vK3OfL6ImZijfLM8RZLtdGo-eWYQaX36zAI45wNB_sEUrXbfKAr7yGxa970ZBX3xpCpBLdGQBdsTY5Grw9iAzuS0P6QUBMFcyE757PMyLRkHJWDVVErDo3CvekdN09WKQPYQF4eFleF4IL9xg1hPBT9RXjlYvd-_Wi6Z_hwpOaYWHt3sDzpQfghC41tXKlhUEfrW-trRfloC0NllWoLMKHLmGs-LEeNcWhy1joXDrvrxus2WCrojYUE7jEyeVBTB3FAE4BZEH9SABGHKXa65VoCHo&state=Rln8tKCmNzuupdrpR1ne849MU-cXKPNpzf95CV6307E%3a%2ffavicon.ico&session_state=4c9c886f-d322-453e-9719-26a4d5196ee7" HTTP/1.1 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" 500 2836 0.161
10.244.5.11:55352 - 424ac2256f5ab031e251d211ac696be7 - - [2024/02/27 10:31:49] ide.software-machines.online GET / "/" HTTP/1.1 "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com" 200 173 0.006

Additional context

Azure

maheshrajrp commented 8 months ago

@KTzerras You have sensitive information here which needs to be redacted ASAP (Better would be create new issue). Can you share screenshot of the same, I went through the exact process a week or two ago ? I might be able to help.

tolusha commented 8 months ago

@KTzerras

whereas we used the file che-cluster-patch.yaml with the following content:

Agree, the oAuthSecret field must be the following: oAuthSecret: $(az ad app credential reset --id $(az ad app list --query "[?displayName=='$ECLIPSE_CHE_APPLICATION_DISPLAY_NAME'].id" --output tsv) --query "password" --output tsv)

Other than that, I successfully followed the doc and installed EclipseChe on Azure AKS. So, it hard to say what caused the problem above.

KTzerras commented 8 months ago

@maheshrajrp

ok, we have hidden the sensitive information.

Which screenshots do you mean exactly?

For the first I attach a screenshot of the file che-cluster-patch.yaml:

che_cluster_patch_screenshot

KTzerras commented 8 months ago

@tolusha After applying file che-cluster-patch.yaml being corrected according to your typo improvement it still does not work and delivers the same error.

KTzerras commented 8 months ago

After a discussion with Microsoft Azure Support the following root cause has been identified:

Since 15.11.2018 Azure AD stopped accepting previously used authentication codes for apps (c,f, https://learn.microsoft.com/en-us/entra/identity-platform/reference-breaking-changes#authorization-codes-can-no-longer-be-reused). Eclipse Che seems though that it still re-uses authentication codes to get tokens for multiple resources. And this leads to our error. According to this, an adaptation of Eclipse Che would be needed that shall a) use the authentication code to get a refresh token b) use that refresh token to acquire additional tokens for other resources

What do you think? Is this the case?

maheshrajrp commented 8 months ago

Hi @KTzerras , I think Eclipse Che follows this way. In any case, we were able to deploy so this flow is already working. My guess is probably some misconfiguration on your side.

"Access token validation failure. Invalid audience." This error seems to be with the authentication part of flow. Can you share a video or something that helps understand when this exception happens, more like the flow. It might help debug better.

KTzerras commented 8 months ago

Dear @maheshrajrp , so you mean you have managed to successfully deploy and use Eclipse Che on Azure Cloud? If yes, did you follow hereby the instructions from https://eclipse.dev/che/docs/stable/administration-guide/installing-che-on-microsoft-azure/ ? Or did you follow another approach? Thank you for your support, I really appreciate it ...

maheshrajrp commented 8 months ago

Yes, I followed the guide and was working for me.

Regarding the Linux Variable Issue, I work with Windows, so had to query and patch the values.yaml manually, so didn't encounter the variable issue. Aside from that, I believe to have followed only those mentioned step. We followed those steps and kept the deployment until we realized Azure Entra B2B Cost Pricing didn't justify the work we did hence had to try a different approach as mentioned here. Feel free to check that out if it matches your usecase.

https://github.com/eclipse/che/issues/22845

KTzerras commented 8 months ago

What do you mean with "Linux Variable Issue"? (I also work with Windows)

maheshrajrp commented 8 months ago

I was pointing to this Typo Issue. https://github.com/eclipse-che/che-docs/pull/2695/commits/9991155f159140a96e6a10b9312bf341e4811a0b

KTzerras commented 8 months ago

@maheshrajrp Your use case does not really match our needs. We want to deploy and run Eclipse Che on the Azure Cloud and are ok with the Cost Pricing. I plan to repeat all steps from scratch and provide a precise description based on which we can then discuss possible solutions,.

KTzerras commented 8 months ago

I close it and shall open a new ticket later if needed.

netomi commented 4 months ago

@KTzerras you provided a Azure Active Directory Application Secret as part of your comment. I would highly suggest you to revoke or change it if its still active.

KTzerras commented 3 months ago

Yes I did within my desperate tries to install Eclipse Che on an Azure Cloud based on the idiotic official documentation. All these attampts were in vain, so we decided to abandon Eclipe Che from our company plans and take Eclipse Theia instead. So the cluster within which I provided the secret is history. With best regards Kostas