search

jupyterhub / binderhub

Run your code in the cloud, with technology so advanced, it feels like magic!
https://binderhub.readthedocs.io
BSD 3-Clause "New" or "Revised" License
2.56k stars 390 forks source link

Update to z2jh 2 and JupyterHub 3 #1544

Closed consideRatio closed 2 years ago

consideRatio commented 2 years ago

This better be done at some point in time, and one reason to do it is because it makes us compatible with k8s 1.25 and higher among other things.

Since this may be relevant knowing about I pinged a lot of you who I guessed could be influenced.

Draft status of PR because of ...

[D 2022-10-10 14:15:14.238 JupyterHub app:2035] Loading roles into database
[I 2022-10-10 14:15:14.249 JupyterHub roles:173] Role jupyterhub-idle-culler added to database
[I 2022-10-10 14:15:14.251 JupyterHub app:1934] Not using allowed_users. Any authenticated user will be allowed.
[W 2022-10-10 14:15:14.264 JupyterHub app:2312] Service binder sets `admin: True`, which is deprecated in JupyterHub 2.0. You can assign now assign roles via `JupyterHub.load_roles` configuration. If you specify services in the admin role configuration, the Service admin flag will be ignored.
[I 2022-10-10 14:15:14.265 JupyterHub roles:238] Adding role admin for Service: binder
[E 2022-10-10 14:15:14.268 JupyterHub app:3297]
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 3294, in launch_instance_async
        await self.initialize(argv)
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 2824, in initialize
        self.init_services()
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/app.py", line 2338, in init_services
        setattr(service, key, value)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 715, in __set__
        self.set(obj, value)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 689, in set
        new_value = self._validate(obj, value)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 723, in _validate
        value = self._cross_validate(obj, value)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 729, in _cross_validate
        value = obj._trait_validators[self.name](obj, proposal)
      File "/usr/local/lib/python3.9/site-packages/traitlets/traitlets.py", line 1132, in __call__
        return self.func(*args, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/jupyterhub/services/service.py", line 322, in _validate_client_id
        raise ValueError(
    ValueError: service  has oauth_client_id='binder-oauth-client-test'. Service oauth client ids must start with 'service-'

[D 2022-10-10 14:15:14.270 JupyterHub application:961] Exiting application: jupyterhub

And failures following trialing prefixing with service-. Feel free to push commits to resolve this!

consideRatio commented 2 years ago

JupyterHub logs of high relevance

[E 2022-10-10 14:34:49.702 JupyterHub auth:271] User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder
[W 2022-10-10 14:34:49.703 JupyterHub web:[179](https://github.com/jupyterhub/binderhub/actions/runs/2987255039/jobs/4789548250#step:28:182)6] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (::ffff:10.1.1.85): You do not have permission to access JupyterHub service binder

Additional log lines for more context

[W 2022-10-10 14:34:45.684 JupyterHub app:2312] Service binder sets `admin: True`, which is deprecated in JupyterHub 2.0. You can assign now assign roles via `JupyterHub.load_roles` configuration. If you specify services in the admin role configuration, the Service admin flag will be ignored.
[I 2022-10-10 14:34:45.685 JupyterHub roles:238] Adding role admin for Service: binder
[I 2022-10-10 14:34:45.691 JupyterHub provider:651] Creating oauth client service-binder
[I 2022-10-10 14:34:45.706 JupyterHub app:2244] Adding API token for service: binder
[D 2022-10-10 14:34:45.712 JupyterHub app:2274] Purging expired APITokens
[D 2022-10-10 14:34:45.713 JupyterHub app:2274] Purging expired OAuthCodes
[D 2022-10-10 14:34:45.716 JupyterHub app:2110] Loading role assignments from config
[W 2022-10-10 14:34:45.729 JupyterHub app:2657] Allowing service binder to complete OAuth without confirmation on an authorization web page
...
[I 2022-10-10 14:34:49.605 JupyterHub log:186] 302 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] -> /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dservice-binder%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (@::ffff:10.1.1.85) 1.39ms
[I 2022-10-10 14:34:49.650 JupyterHub log:186] 200 GET /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dservice-binder%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (@::ffff:10.1.1.85) 33.09ms
[D 2022-10-10 14:34:49.662 JupyterHub roles:281] Assigning default role to User dummy
[I 2022-10-10 14:34:49.667 JupyterHub roles:238] Adding role user for User: dummy
[D 2022-10-10 14:34:49.672 JupyterHub roles:281] Assigning default role to User dummy
[D 2022-10-10 14:34:49.676 JupyterHub base:559] Setting cookie jupyterhub-session-id: {'httponly': True, 'path': '/'}
[D 2022-10-10 14:34:49.676 JupyterHub base:563] Setting cookie for dummy: jupyterhub-hub-login
[D 2022-10-10 14:34:49.676 JupyterHub base:559] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2022-10-10 14:34:49.677 JupyterHub base:810] User logged in: dummy
[I 2022-10-10 14:34:49.677 JupyterHub log:186] 302 POST /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dservice-binder%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ -> /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] (dummy@::ffff:10.1.1.85) 18.70ms
[D 2022-10-10 14:34:49.690 JupyterHub base:275] Recording first activity for <User(dummy 0/1 running)>
[D 2022-10-10 14:34:49.694 JupyterHub provider:415] Validating client id service-binder
[D 2022-10-10 14:34:49.695 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:363] Validating redirection uri http://127.0.0.1:8585/oauth_callback for client service-binder.
[D 2022-10-10 14:34:49.695 oauthlib.oauth2.rfc6749.grant_types.base base:230] Using provided redirect_uri http://127.0.0.1:8585/oauth_callback
[D 2022-10-10 14:34:49.700 JupyterHub provider:490] validate_redirect_uri: client_id=service-binder, redirect_uri=http://127.0.0.1:8585/oauth_callback
[D 2022-10-10 14:34:49.701 oauthlib.oauth2.rfc6749.grant_types.base base:171] Validating access to scopes ['access:services!service=binder', 'read:users:name!user', 'read:users:groups!user'] for client 'service-binder' (<OAuthClient(identifier='service-binder')>).
[D 2022-10-10 14:34:49.702 JupyterHub provider:614] Allowing request for scope(s) for service-binder:  read:users:groups!user,access:services!service=binder,read:users:name!user
[E 2022-10-10 14:34:49.702 JupyterHub auth:271] User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder
[W 2022-10-10 14:34:49.703 JupyterHub web:[179](https://github.com/jupyterhub/binderhub/actions/runs/2987255039/jobs/4789548250#step:28:182)6] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (::ffff:10.1.1.85): You do not have permission to access JupyterHub service binder

Origin of error message

https://github.com/jupyterhub/jupyterhub/blob/5997614f452a3cfebfa541800012a914d37f3ec8/jupyterhub/apihandlers/auth.py#L265-L283

consideRatio commented 2 years ago

I think I've pinpointed the differences of relevance here.

The different outcome is determined in the GET webrequest handler for JupyterHub's /api/oauth2/authorize path, specifically this code segment from jupyterhub 3.0.0 and this code segment from jupyterhub 1.5.0.

In 3.0.0 we get User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder, while in 1.5.0 we get Skipping oauth confirmation for <User(dummy 0/1 running)> accessing JupyterHub service binder. The JupyterHub registered service binder is configured with oauth_no_confirm: true, and that is why we observe the "Skipping oauth confirmation" message. Should we do that also for 3.0.0? If so, that logic maybe needs to be given higher priority in JupyterHub.

With z2jh 2.0.0 and JupyterHub 3.0.0

[I 2022-10-10 14:34:49.677 JupyterHub base:810] User logged in: dummy
[I 2022-10-10 14:34:49.677 JupyterHub log:186] 302 POST /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dservice-binder%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ -> /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] (dummy@::ffff:10.1.1.85) 18.70ms
[D 2022-10-10 14:34:49.690 JupyterHub base:275] Recording first activity for <User(dummy 0/1 running)>
[D 2022-10-10 14:34:49.694 JupyterHub provider:415] Validating client id service-binder
[D 2022-10-10 14:34:49.695 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:363] Validating redirection uri http://127.0.0.1:8585/oauth_callback for client service-binder.
[D 2022-10-10 14:34:49.695 oauthlib.oauth2.rfc6749.grant_types.base base:230] Using provided redirect_uri http://127.0.0.1:8585/oauth_callback
[D 2022-10-10 14:34:49.700 JupyterHub provider:490] validate_redirect_uri: client_id=service-binder, redirect_uri=http://127.0.0.1:8585/oauth_callback
[D 2022-10-10 14:34:49.701 oauthlib.oauth2.rfc6749.grant_types.base base:171] Validating access to scopes ['access:services!service=binder', 'read:users:name!user', 'read:users:groups!user'] for client 'service-binder' (<OAuthClient(identifier='service-binder')>).
[D 2022-10-10 14:34:49.702 JupyterHub provider:614] Allowing request for scope(s) for service-binder:  read:users:groups!user,access:services!service=binder,read:users:name!user
[E 2022-10-10 14:34:49.702 JupyterHub auth:271] User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder
[W 2022-10-10 14:34:49.703 JupyterHub web:[179](https://github.com/jupyterhub/binderhub/actions/runs/2987255039/jobs/4789548250#step:28:182)6] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjE2MmM1NzNjMDJmMDRlYmE4OGQ4ZjNiODM1NTZlZTM5IiwgIm5leHRfdXJsIjogIi8ifQ (::ffff:10.1.1.85): You do not have permission to access JupyterHub service binder

With z2jh 1.2.0 and JupyterHub 1.5.0

[I 2022-10-12 10:31:27.419 JupyterHub base:762] User logged in: dummy
[I 2022-10-12 10:31:27.419 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2Fapi%2Foauth2%2Fauthorize%3Fclient_id%3Dbinder-oauth-client-test%26redirect_uri%3Dhttp%253A%252F%252F127.0.0.1%253A8585%252Foauth_callback%26response_type%3Dcode%26state%3DeyJ1dWlkIjogIjZlYmM2YzgyZDk1ODQ2MjNhMmU4ZGNhOWFjYjJhNTg2IiwgIm5leHRfdXJsIjogIi8ifQ -> /hub/api/oauth2/authorize?client_id=binder-oauth-client-test&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] (dummy@::ffff:10.1.0.155) 16.90ms
[D 2022-10-12 10:31:27.429 JupyterHub base:283] Recording first activity for <User(dummy 0/1 running)>
[D 2022-10-12 10:31:27.431 JupyterHub provider:409] Validating client id binder-oauth-client-test
[D 2022-10-12 10:31:27.437 JupyterHub provider:484] validate_redirect_uri: client_id=binder-oauth-client-test, redirect_uri=http://127.0.0.1:8585/oauth_callback
[D 2022-10-12 10:31:27.439 JupyterHub auth:252] Skipping oauth confirmation for <User(dummy 0/1 running)> accessing JupyterHub service binder
[D 2022-10-12 10:31:27.439 JupyterHub provider:409] Validating client id binder-oauth-client-test
[D 2022-10-12 10:31:27.440 JupyterHub provider:484] validate_redirect_uri: client_id=binder-oauth-client-test, redirect_uri=http://127.0.0.1:8585/oauth_callback
[D 2022-10-12 10:31:27.440 JupyterHub provider:231] Saving authorization code binder-oauth-client-test, uPo..., (), {}
[I 2022-10-12 10:31:27.444 JupyterHub log:189] 302 GET /hub/api/oauth2/authorize?client_id=binder-oauth-client-test&redirect_uri=http%3A%2F%2F127.0.0.1%3A8585%2Foauth_callback&response_type=code&state=[secret] -> http://127.0.0.1:8585/oauth_callback?code=[secret]&state=[secret] (dummy@::ffff:10.1.0.155) 15.75ms
consideRatio commented 2 years ago

:tada: wiee nice @minrk I see the tests passing with your changes!

Sorry for missing to bump the jupyterhub version, we even had a note about it... DOH!

minrk commented 2 years ago

I believe the missing bit is that users must be granted access to services explicitly via load_roles.

That's the

User <User(dummy 0/1 running)> not allowed to access JupyterHub service binder

message, which translates to "user dummy does not have scope access:services!binder." Granting users access to services via loadRoles should get past this step, at least. I added access:services to the user role.

JupyterHub in the binderhub image also needs to be updated to match, which I added as well.

The binder service also should not need to be admin, we should define Binder's own permissions in a role. It needs:

  1. servers to read, start, and stop user servers
  2. admin:users (if not using auth)

It may make sense to define the role(s) dynamically in Python, because of the if auth_enabled logic involved. Or we could define two roles statically in config, and then assign the role dynamically, depending on if auth is enabled or not.

Alternately, it could get no permissions at all in the auth case, and we could perform the launch requests with the user's own token. That might be better! It would mean specifying the permission servers!user in the services.binder.oauth_client_allowed_scopes config. It would still mean defining the (default) binder role (or not) and then assigning it (or not) based on auth_enabled.

consideRatio commented 2 years ago

@minrk I opened https://github.com/jupyterhub/binderhub/issues/1553 to represent your comments suggestions on how to tighten the requested permissions needed.

Thanks for completing this PR, this LGTM!

manics commented 2 years ago

Do we need to update repo2docker to match the JupyterHub bump? https://github.com/jupyterhub/repo2docker/blob/d7be04efb590a2a1227614ccaf9b504ad9333470/repo2docker/buildpacks/conda/environment.yml#L11

consideRatio commented 2 years ago

Yepp, we need a new jupyterhub version for compatibility i think :/

manics commented 2 years ago

I think we should make one last release before bumping it: https://github.com/jupyterhub/repo2docker/pull/1194