JupyterHub demos are broken

[jhamrick]

With JupyterHub 2.0, the demos that use services (i.e. demo_one_class_multiple_graders and demo_multiple_classes) no longer work in that the instructors do not have permission to access the formgrader. I thought it would be a matter of simply adding something like the following (for demo_one_class_multiple_graders) to jupyterhub_config.py:

c.JupyterHub.load_roles = [
    {
        'name': 'formgrader-course101',
        'groups': ['formgrader-course101'],
        'scopes': [
            'access:services!service=course101',
        ]

    }
]

but this doesn't seem to work; I get an error message that [D 2022-06-08 19:38:28.329 SingleUserNotebookApp auth:995] Hub user instructor1 needs scope(s) {'access:services', 'access:services!service=course101'}, has scope(s) ['access:servers!user=instructor1', 'read:users:activity!user=instructor1', 'users:activity!user=instructor1']. This is true also if I use 'users': ['instructor1'] rather than 'groups'. I am not sure why the scopes are not being correctly set.

This is related to #1533, which documents that there is an incorrect scope configuration for the formgrader service itself too.

@perllaghu do you have any insight on this from your deployment of nbgrader?

[perllaghu]

@perllaghu do you have any insight on this from your deployment of nbgrader?

I'm not seeing any errors like this, however we dropped use of jupyterhub some time ago, and spawn notebooks in docker images directly into kubernetes from our external service management code, and our own exchange get_current_user authenticator.

[jhamrick]

Ah ok, I didn't realize you weren't using JupyterHub!

Perhaps @minrk might be able to help?

[perllaghu]

Ah ok, I didn't realize you weren't using JupyterHub!

We found that the jupyterhub proxy (linking users to their spawned notebooks) was getting slower as the number of simultanious users increased, and badly failed if the hub crashed

[minrk]

@jhamrick I was able to run the jupyterhub demo, and the permissions do seem to work with exactly your config. The only thing I could find is that there's a typo in your load_roles, where the load_groups in the demo has formgrade-course101 (no 'r') while your load_roles has formgrader-course101. This results in the (not loud enough and should probably be an error) log message:

[I 2022-06-09 11:41:14.284 JupyterHub app:2151] Found unexisting groups formgrader-course101 in role definition formgrader-course101

Making the group name match gives the appropriate group members access to the service.

@perllaghu I certainly don't want to push folks to JupyterHub who have found something that's a better fit, but I'm curious about the scale and activity when you had performance issues. We do now have a horizontally scalable proxy implementation with traefik, but have found that the default single process configurable-http-proxy has never been a performance bottleneck and comfortably handles at least several hundred concurrent users, given sufficient resources.

[perllaghu]

@perllaghu I certainly don't want to push folks to JupyterHub who have found something that's a better fit, but I'm curious about the scale and activity when you had performance issues. We do now have a horizontally scalable proxy implementation with traefik, but have found that the default single process configurable-http-proxy has never been a performance bottleneck and comfortably handles at least several hundred concurrent users, given sufficient resources.

Likewise, I'm interested in what other people have.

We seem to run about a steady 300-400 notebooks during term time, with the occasional peaks trying to double that, and run everything in a k8 cluster (35 nodes, 136 cores, 800GiB Memory.... and, likewise, now use traefik)

Perhaps there'd be interest in a [virtual] get-together to see how/why people have the services they have

[jhamrick]

@minrk Hmm strange, I don't have that typo and when I try to access the formgrader via the instructor accounts it doesn't work. This is the full config I'm using:

c = get_config()

# Our user list
c.Authenticator.allowed_users = [
    'instructor1',
    'instructor2',
    'student1',
]

# instructor1 and instructor2 have access to a shared server:
c.JupyterHub.load_groups = {
    'formgrader-course101': [
        'instructor1',
        'instructor2'
    ]
}
c.JupyterHub.load_roles = [
    {
        'name': 'formgrader-course101',
        'groups': ['formgrader-course101'],
        'scopes': [
            'access:services!service=course101',
        ]

    }
]

# Start the notebook server as a service. The port can be whatever you want
# and the group has to match the name of the group defined above. The name of
# the service MUST match the name of your course.
c.JupyterHub.services = [
    {
        'name': 'course101',
        'url': 'http://127.0.0.1:9999',
        'command': [
            'jupyterhub-singleuser',
            '--debug',
        ],
        'user': 'grader-course101',
        'cwd': '/home/grader-course101'
    }
]

Then when I log in as instructor1 and go to the "Courses" tab, it says "There are no available formgrader services." and the error message I pasted above can be found in the terminal. But for you that works and you can access the formgrader? 🤔

[minrk]

I was able to access formgrader. Testing again with that exact config. Can you share logs from the service startup?

[minrk]

I just ran it again with your config unmodified, and it's working.

I followed the restart_demo.sh script to set up a fresh Ubuntu 20.04 VM (a couple minor tweaks, because pip install -e didn't seem to work, but pip install . did)

My env, if it helps:

pip freeze

``` alembic==1.8.0 anyio==3.6.1 argon2-cffi==21.3.0 argon2-cffi-bindings==21.2.0 asttokens==2.0.5 async-generator==1.10 attrs==19.3.0 Automat==0.8.0 backcall==0.2.0 beautifulsoup4==4.11.1 bleach==5.0.0 blinker==1.4 certifi==2019.11.28 certipy==0.1.3 cffi==1.15.0 chardet==3.0.4 charset-normalizer==2.0.12 Click==7.0 cloud-init==22.2 colorama==0.4.3 command-not-found==0.3 configobj==5.0.6 constantly==15.1.0 cryptography==2.8 dbus-python==1.2.16 debugpy==1.6.0 decorator==5.1.1 defusedxml==0.7.1 distro==1.4.0 distro-info===0.23ubuntu1 entrypoints==0.3 executing==0.8.3 fastjsonschema==2.15.3 greenlet==1.1.2 httplib2==0.14.0 hyperlink==19.0.0 idna==2.8 importlib-metadata==1.5.0 importlib-resources==5.7.1 incremental==16.10.1 ipykernel==6.13.1 ipython==8.4.0 ipython-genutils==0.2.0 ipywidgets==7.7.0 jarowinkler==1.0.2 jedi==0.18.1 Jinja2==3.1.2 jsonpatch==1.22 jsonpointer==2.0 jsonschema==3.2.0 jupyter-client==7.3.4 jupyter-core==4.10.0 jupyter-server==1.17.1 jupyter-telemetry==0.1.0 jupyterhub==2.3.1 jupyterlab-pygments==0.2.2 jupyterlab-widgets==1.1.0 keyring==18.0.1 language-selector==0.1 launchpadlib==1.10.13 lazr.restfulclient==0.14.2 lazr.uri==1.0.3 Mako==1.2.0 MarkupSafe==2.1.1 matplotlib-inline==0.1.3 mistune==0.8.4 more-itertools==4.2.0 nbclient==0.6.4 nbconvert==6.5.0 nbformat==5.4.0 nbgrader==0.8.0.dev0 nest-asyncio==1.5.5 netifaces==0.10.4 notebook==6.4.12 oauthlib==3.1.0 packaging==21.3 pamela==1.0.0 pandocfilters==1.5.0 parso==0.8.3 pexpect==4.6.0 pickleshare==0.7.5 prometheus-client==0.14.1 prompt-toolkit==3.0.29 psutil==5.9.1 ptyprocess==0.7.0 pure-eval==0.2.2 pyasn1==0.4.2 pyasn1-modules==0.2.1 pycparser==2.21 Pygments==2.12.0 PyGObject==3.36.0 PyHamcrest==1.9.0 PyJWT==1.7.1 pymacaroons==0.13.0 PyNaCl==1.3.0 pyOpenSSL==19.0.0 pyparsing==3.0.9 pyrsistent==0.15.5 pyserial==3.4 python-apt==2.0.0+ubuntu0.20.4.7 python-dateutil==2.8.2 python-debian===0.1.36ubuntu1 python-json-logger==2.0.2 PyYAML==5.3.1 pyzmq==23.1.0 qtconsole==5.3.1 QtPy==2.1.0 rapidfuzz==2.0.11 requests==2.27.1 requests-unixsocket==0.2.0 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.6 SecretStorage==2.3.1 Send2Trash==1.8.0 service-identity==18.1.0 simplejson==3.16.0 six==1.14.0 sniffio==1.2.0 sos==4.3 soupsieve==2.3.2.post1 SQLAlchemy==1.4.37 ssh-import-id==5.10 stack-data==0.2.0 systemd-python==234 terminado==0.15.0 tinycss2==1.1.1 tornado==6.1 traitlets==5.1.1 Twisted==18.9.0 ubuntu-advantage-tools==27.8 ufw==0.36 unattended-upgrades==0.1 urllib3==1.25.8 wadllib==1.3.3 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.2 widgetsnbextension==3.6.0 zipp==3.8.0 zope.interface==4.7.1 ```

Can you provide a more detailed log dump?

[jhamrick]

Huh, I am also using a new VM I just created yesterday (though I am using Ubuntu 22.04). Just to be safe I tried uninstalling and reinstalling JupyterHub and Jupyter, but no difference. The pip install command in the demo works fine for me though; the only changes I had to make was to do git checkout main rather than git checkout master, and to add mkdir -p /etc/jupyter before copying the global nbgrader config. Did you make any other tweaks to the demo scripts?

Here's the JupyterHub log output and output from pip freeze: demo.log pip.log

Thanks so much for helping me debug this!

[minrk]

I'll try again in the morning and capture more logs. Maybe even try to encapsulate it in docker so it's portable.

I did set the dummy authenticator, but that shouldn't affect anything after login: https://github.com/jupyter/nbgrader/compare/main...minrk:debug . I also switched the jupyter install to notebook.

[jhamrick]

Thanks for sending your changes!

So one thing I found is that if I navigate to the formgrader directly, i.e. going to /services/course101/formgrader, then I can access it as well---but it doesn't show up in the course list extension and that's where I'm getting that error message. So it is a problem with the course list extension that for some reason it thinks it doesn't have access to the formgrader. Will continue to investigate...

[jhamrick]

For reference, this is the function where it's failing the authorization to see the formgrader: https://github.com/jupyter/nbgrader/blob/main/nbgrader/server_extensions/course_list/handlers.py#L90

[minrk]

Aha! I suspect that's an issue with the permission of the server token, which makes the API request to the Hub from the server extension. The server's API token has significantly reduced permissions by default. I'll look into a fix in #1597

[minrk]

The course list seems to work now in #1597

What should the student see in these demos? I don't see anything, but I don't know if they should.

[jhamrick]

Amazing, thanks so much! We can continue the conversation over in https://github.com/jupyter/nbgrader/pull/1597...