search

jupyterhub / oauthenticator

OAuth + JupyterHub Authenticator = OAuthenticator
https://oauthenticator.readthedocs.io
BSD 3-Clause "New" or "Revised" License
409 stars 363 forks source link

[Generic] v16 undocumented change - client_id / client_secret no longer passed in request to token_url when basic_auth is True #646

Closed dmpe closed 1 year ago

dmpe commented 1 year ago

Bug description

After trying 3.0.0-beta.2.git.6228.hfe344e7a I am getting an issue whereby it is impossible to login into JHub using keycloak and get e.g. admin page properly displayed. It works great with 2.0.0 - 09 September 2022 - 3.0.0 version of the helm chart and it works also when using 3.0.0-beta.1. But with recently released beta 2 it has stopped working for me, at least.

And unfortunately, keycloak is my only option to use.

Expected behaviour

Login with keycloak works again :)

Actual behaviour

After logging with keycloak, I am being redirected to Jupyterhub which shows 500 : Internal Server Error

How to reproduce

Following helm chart config is being used:

hub:
  config:
    GenericOAuthenticator:
      authorize_url: https://keycloak.firm.com/auth/realms/Jupyter/protocol/openid-connect/auth
      token_url: https://keycloak.firm.com/auth/realms/Jupyter/protocol/openid-connect/token
      userdata_url: https://keycloak.firm.com/auth/realms/Jupyter/protocol/openid-connect/userinfo
      enable_auth_state: true
      client_id: {{ .Values.jupyterhub.auth.generic.clientId }}
      client_secret: {{ .Values.jupyterhub.auth.generic.clientSecret }}
      oauth_callback_url: 'https://jupyter.firm.com/hub/oauth_callback'
      login_service: Keycloak -> This works now well, see https://github.com/jupyterhub/oauthenticator/issues/643
      allow_existing_users: true
      scope:
        - openid
        - profile
        - roles
      username_claim: preferred_username # or username_key
      userdata_params:
        state: state
      admin_users:
        - admin1
    JupyterHub:
      authenticator_class: generic-oauth
    CryptKeeper:
      keys:
       - {{ .Values.jupyterhub.hub.cookieSecret }}

{{ .Values.jupyterhub.auth.generic.clientId }}, etc. come from Mozilla SOPS and I have 100% verified that when deploying helm chart to k8s, the secret hub contains correct clientID and clientSecret values, filled with secrets that are stored in SOPS file.

Your personal set up

Full environment
Logs Jhub shows following ``` [D 2023-07-06 08:19:45.418 JupyterHub proxy:392] Checking routes [I 2023-07-06 08:19:45.418 JupyterHub proxy:477] Adding route for Hub: / => http://hub:8081 [D 2023-07-06 08:19:45.418 JupyterHub proxy:880] Proxy: Fetching POST http://proxy-api:8001/api/routes/ [I 2023-07-06 08:19:45.421 JupyterHub app:3247] JupyterHub is now running, internal Hub API at http://hub:8081/hub/ [D 2023-07-06 08:19:45.422 JupyterHub app:2852] It took 0.597 seconds for the Hub to start [D 2023-07-06 08:19:45.984 JupyterHub base:297] Recording first activity for [I 2023-07-06 08:19:45.991 JupyterHub log:191] 200 GET /hub/api/ (jupyterhub-idle-culler@127.0.0.1) 8.62ms [D 2023-07-06 08:19:45.995 JupyterHub scopes:863] Checking access via scope list:users [D 2023-07-06 08:19:45.995 JupyterHub scopes:677] Unrestricted access to /hub/api/users via list:users [I 2023-07-06 08:19:46.000 JupyterHub log:191] 200 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@127.0.0.1) 6.97ms [D 2023-07-06 08:20:45.421 JupyterHub proxy:880] Proxy: Fetching GET http://proxy-api:8001/api/routes [D 2023-07-06 08:20:45.426 JupyterHub proxy:392] Checking routes [D 2023-07-06 08:20:47.466 JupyterHub log:191] 200 GET /hub/health (@10.50.16.15) 0.77ms [D 2023-07-06 08:20:47.942 JupyterHub pages:584] No template for 503 [I 2023-07-06 08:20:47.975 JupyterHub log:191] 200 GET /hub/error/503?url=%2F (@10.244.201.8) 44.19ms [D 2023-07-06 08:20:57.466 JupyterHub log:191] 200 GET /hub/health (@10.50.16.15) 0.69ms [I 2023-07-06 08:21:03.499 JupyterHub log:191] 302 GET /hub/home -> /hub/login?next=%2Fhub%2Fhome (@10.50.16.15) 0.73ms [I 2023-07-06 08:21:03.531 JupyterHub log:191] 200 GET /hub/login?next=%2Fhub%2Fhome (@10.50.16.15) 9.38ms [I 2023-07-06 08:21:04.752 JupyterHub oauth2:102] OAuth redirect: https://jupyter.firm.com/hub/oauth_callback [D 2023-07-06 08:21:04.753 JupyterHub base:585] Setting cookie oauthenticator-state: {'httponly': True, 'secure': True, 'expires_days': 1} [I 2023-07-06 08:21:04.759 JupyterHub log:191] 302 GET /hub/oauth_login?next=%2Fhub%2Fhome -> https://keycloak.firm.com/auth/realms/Jupyter/protocol/openid-connect/auth?response_type=code&redirect_uri=https%3A%2F%2Fjupyter.firm.com%2Fhub%2Foauth_callback&client_id=jupyterhub&state=[secret]&scope=openid+profile+roles (@10.50.16.15) 6.90ms [E 2023-07-06 08:21:06.178 JupyterHub oauth2:596] Error fetching 401 POST https://keycloak.firm.com/auth/realms/Jupyter/protocol/openid-connect/token: { "error": "invalid_client", "error_description": "Invalid client or Invalid client credentials" } [E 2023-07-06 08:21:06.179 JupyterHub web:1871] Uncaught exception GET /hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJiMWU3N2EwMWE3ODY0N2E3YWE1ZjgzNzUyNGZkYzczNSIsICJuZXh0X3VybCI6ICIvaHViL2hvbWUifQ%3D%3D&session_state=7dc7a0f8-10cd-4a52-8de4-ac5d13185af4&code=25e396ef-2a48-4f55-9286-fb61d7926805.7dc7a0f8-10cd-4a52-8de4-ac5d13185af4.189bf5c7-e269-4b29-b7f3-a85dad715d21 (10.50.16.15) HTTPServerRequest(protocol='https', host='jupyter.firm.com', method='GET', uri='/hub/oauth_callback?state=eyJzdGF0ZV9pZCI6ICJiMWU3N2EwMWE3ODY0N2E3YWE1ZjgzNzUyNGZkYzczNSIsICJuZXh0X3VybCI6ICIvaHViL2hvbWUifQ%3D%3D&session_state=7dc7a0f8-10cd-4a52-8de4-ac5d13185af4&code=25e396ef-2a48-4f55-9286-fb61d7926805.7dc7a0f8-10cd-4a52-8de4-ac5d13185af4.189bf5c7-e269-4b29-b7f3-a85dad715d21', version='HTTP/1.1', remote_ip='10.50.16.15') Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/tornado/web.py", line 1786, in _execute result = await result ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 202, in get user = await self.login_user() ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/jupyterhub/handlers/base.py", line 826, in login_user authenticated = await self.authenticate(data) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/jupyterhub/auth.py", line 492, in get_authenticated_user authenticated = await maybe_future(self.authenticate(handler, data)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 932, in authenticate token_info = await self.get_token_info(handler, access_token_params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 806, in get_token_info token_info = await self.httpfetch( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 631, in httpfetch return await self.fetch( ^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 597, in fetch raise e File "/usr/local/lib/python3.11/site-packages/oauthenticator/oauth2.py", line 576, in fetch resp = await self.http_client.fetch(req, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.httpclient.HTTPClientError: HTTP 401: Unauthorized ```
consideRatio commented 1 year ago

After logging with keycloak, I am being redirected to Jupyterhub which shows 500 : Internal Server Error

What does the jupyterhub logs say? EDIT: ooops they are part of the issue!!!

consideRatio commented 1 year ago

Is it correct that your OAuth client id registered via keycloak is jupyterhub?

consideRatio commented 1 year ago

I don't get why someting goes wrong yet, but its clear to me that it goes wrong quite early.

  1. The authorize_url is successfully accessed, and the user who was redirected to it arrives back to the callback/redirect URL with a code
  2. The code now meant to be used with client_id in a request to token_url to receive an access token, but this request fails with a JSON response like:
{
    "error": "invalid_client",
     error_description": "Invalid client or Invalid client credentials"
}
consideRatio commented 1 year ago

@dmpe can you try configuring GenericOAuthenticator.basic_auth: true?

consideRatio commented 1 year ago

Oh sorry, try it set to false instead I think.

consideRatio commented 1 year ago

Technical details summarized

basic_auth function

In 15.1.0, only the GenericOAuthenticator had a config called basic_auth, and it defaulted to True. When it was True, the client_id and client_secret was encoded in a HTTP header in the request to the token_url.

basic_auth moved to the OAuthenticator base class

The basic_auth config was moved to the OAuthnticator base class. This change is probably entirely unrelated to the issue observed. In the base class it was disabled by default, and enabled by default specifically for GenericOAuthenticator.

What changed

There is one change for GenericOAuthenticator users that had basic_auth enabled, which all GenericOAuthenticator users have unless they have it explicitly set to false. The change is to not post client_id and client_secret also in the post body, but only provide them via the HTTP Basic authentication header when that is done - as it is when basic_auth is True.

In practice, 16.0.0 made GenericOAuthenticator stop using both HTTP Basic authentication and form based authentication, to only use one or the other as suggested in the OAuth2 specification:

Including the client credentials in the request-body using the two parameters is NOT RECOMMENDED and SHOULD be limited to clients unable to directly utilize the HTTP Basic authentication scheme (or other password-based HTTP authentication schemes).

What you should do

I think that you should configure basic_auth to False, and if you do, the client_id and client_secret will once again be passed in the request to token_url, this time without the HTTP Basic authentication header that probably wasn't used.

@dmpe could you verify that setting basic_auth to False works for you? If you can, that is very valuable input to decide what we should do in the oauthenticator project. I figure we need to decide to either revert the previous behavior, document the changed behavior.

dmpe commented 1 year ago

Is it correct that your OAuth client id registered via keycloak is jupyterhub?

that is correct, sorry my previous comment was not precise...

GenericOAuthenticator.basic_auth: true?

tried, does not work.

Tried false and it works just great! Exactly what was missing.

I figure we need to decide to either revert the previous behavior, document the changed behavior.

Yes. either one of them would be welcome.

consideRatio commented 1 year ago

Thank you @dmpe for quickly testing and reporting back!!!

dmpe commented 1 year ago

Thank you very much. Appreciate quick fixes! @consideRatio