search

nebari-dev / nebari

🪴 Nebari - your open source data science platform
https://nebari.dev
BSD 3-Clause "New" or "Revised" License
280 stars 93 forks source link

[ENH] - Move to JupyterHub 5.x (unreleased) #2330

Closed aktech closed 6 months ago

aktech commented 7 months ago

Feature description

We're using JupyterHub 4.0.1 right now and this is a proposal to upgrade JupyterHub from 4.x to 5.x (unreleased).

Steps:

Value and/or benefit

Since the last release of JupyterHub (4.0.2). JupyterHub added the ability to share user servers, which is a crucial feature we would like to harness in Nebari via jhub-apps. Upgrading Nebari to use Jupyterhub 5.x will help us implement app sharing in jhub-apps, for which Nebari is the primary consumer: https://github.com/nebari-dev/jhub-apps/issues/11

We're looking for this feature (which is not available in 4.x):

JupyterHub 5.x release might be around the corner, but I am not sure about the exact timeline right now. We will have to upgrade later anyways, upgrading now will unblock us from using sharing feature.

Anything else?

No response

krassowski commented 7 months ago

An RC for 5.0 might be available soon (next week). As a stepping stone we should migrate to 4.1 which is planned to be released very soon.

krassowski commented 7 months ago

https://github.com/nebari-dev/nebari-docker-images/pull/129 is ready for review.

krassowski commented 7 months ago

As a task for 5.0 migration, we should test setting JUPYTERHUB_ALLOW_TOKEN_IN_URL=0 once we migrated to JupyterHub 4.1 (as this was added in 4.1 and will be a default in 5.0), see https://github.com/nebari-dev/nebari/issues/2351

krassowski commented 6 months ago

Note a new issue tracking JupyterHub 5.0 release: https://github.com/jupyterhub/jupyterhub/issues/4788 - it looks the outstanding work is around migration to Bootstrap 5 and documentation.

krassowski commented 6 months ago

I created a branch on the docker images repo to try out the current state: https://github.com/nebari-dev/nebari-docker-images/tree/jhub-5.0.0.dev though it fails to deploy locally for me with:

[terraform]: │ Error: timed out waiting for the condition
[terraform]: │ 
[terraform]: │   with module.jupyterhub.helm_release.jupyterhub,
[terraform]: │   on modules/kubernetes/services/jupyterhub/main.tf line 54, in resource "helm_release" "jupyterhub":
[terraform]: │   54: resource "helm_release" "jupyterhub" {

I am not sure if this is a problem with my local deployment or because we need to update the helm chart version in:

https://github.com/nebari-dev/nebari/blob/a26b8df19c31363082ca3104e4e7b2e9354b22ae/src/_nebari/stages/kubernetes_services/template/modules/kubernetes/services/jupyterhub/main.tf#L54-L60

There is no pre-release for 5.0 yet: https://hub.jupyter.org/helm-chart/#development-releases-jupyterhub. One think I may try is opening a draft PR forcing this tag to see if deployment passes on CI.

@aktech did you have any success with earlier dev versions of JupyterHub or did you only test jhub-apps in isolation??

aktech commented 6 months ago

@aktech did you have any success with earlier dev versions of JupyterHub or did you only test jhub-apps in isolation??

Yes, tested jhub-apps only in isolation.

it fails to deploy locally for me with:

The problem does seem like it's with the helm chart. Is there are an error in the created pods logs before it crashes?

krassowski commented 6 months ago

Is there are an error in the created pods logs before it crashes?

Yes, it is failing on database migration. I think this is be

│ [I 2024-04-16 14:03:11.075 JupyterHub dbutil:131] Upgrading sqlite:///jupyterhub.sqlite
│ [I 2024-04-16 14:03:11.075 JupyterHub dbutil:100] Backing up jupyterhub.sqlite => jupyterhub.sqlite.2024-04-16-140311
│ [I 2024-04-16 14:03:11.700 alembic.runtime.migration migration:216] Context impl SQLiteImpl.
│ [I 2024-04-16 14:03:11.700 alembic.runtime.migration migration:219] Will assume non-transactional DDL.
│ [E 2024-04-16 14:03:11.706 alembic.util.messaging messaging:73] Can't locate revision identified by '4621fec11365

4621fec11365 is the revision added in my manage_roles PR so I am a bit confused here.

krassowski commented 6 months ago

First beta of JupyterHub 5 is out:

krassowski commented 6 months ago

The docker images for 5.0 beta build nicely: https://github.com/nebari-dev/nebari-docker-images/actions/runs/8780986557

And I pinned to them locally:

default_images:
  jupyterhub: quay.io/nebari/nebari-jupyterhub:jhub-5.0.0
  jupyterlab: quay.io/nebari/nebari-jupyterlab:jhub-5.0.0

But in the hub pod I still see:

│ [I 2024-04-22 11:49:50.665 JupyterHub app:2885] Running JupyterHub version 4.1.0

(and then crash due to failed db migration)

aktech commented 6 months ago

But in the hub pod I still see:

That's because one of the packages is probably requesting jupyterhub 4:

https://github.com/nebari-dev/nebari-docker-images/actions/runs/8780986557/job/24092032511#step:9:1011

Screenshot 2024-04-22 at 1 04 50 pm
aktech commented 6 months ago

I am guessing, pins like these would fetch jupyterhub 4 instead of 5 (since its a release candidate and not a stable release)

https://github.com/jupyterhub/kubespawner/blob/becea538dd075b063964123034111aa6463f79ac/pyproject.toml#L32C1-L33C1

krassowski commented 6 months ago

That's because one of the packages is probably requesting jupyterhub 4:

Yes, but pip uninstalls it later:

Attempting uninstall: jupyterhub Found existing installation: jupyterhub 4.1.5 Uninstalling jupyterhub-4.1.5: Successfully uninstalled jupyterhub-4.1.5 Successfully installed Click-8.1.7 SecretStorage-3.3.3 annotated-types-0.6.0 anyio-4.3.0 argon2-cffi-23.1.0 argon2-cffi-bindings-21.2.0 arrow-1.3.0 asttokens-2.4.1 async-lru-2.0.4 babel-2.14.0 backports.tarfile-1.1.0 beautifulsoup4-4.12.3 bleach-6.1.0 bokeh-3.4.1 bokeh-root-cmd-0.1.2 comm-0.2.2 contourpy-1.2.1 debugpy-1.8.1 decorator-5.1.1 defusedxml-0.7.1 distlib-0.3.8 ecdsa-0.19.0 editables-0.5 exceptiongroup-1.2.1 executing-2.0.1 fastapi-0.110.2 fastjsonschema-2.19.1 filelock-3.13.4 fqdn-1.5.1 h11-0.14.0 hatch-1.9.4 hatchling-1.21.1 httpcore-1.0.5 httpx-0.27.0 hyperlink-21.0.0 ipykernel-6.29.4 ipython-8.18.1 ipywidgets-8.1.2 isoduration-20.11.0 jaraco.classes-3.4.0 jaraco.context-5.3.0 jaraco.functools-4.0.1 jedi-0.19.1 jeepney-0.8.0 jhsingle-native-proxy-0.8.2 jhub-apps-2024.4.1 json5-0.9.25 jsonpointer-2.4 jupyter-1.0.0 jupyter-client-8.6.1 jupyter-console-6.6.3 jupyter-core-5.7.2 jupyter-events-0.10.0 jupyter-lsp-2.2.5 jupyter-server-2.14.0 jupyter-server-terminals-0.5.3 jupyterhub-5.0.0b1 jupyterlab-4.1.6 jupyterlab-pygments-0.3.0 jupyterlab-server-2.26.0 jupyterlab-widgets-3.0.10 keyring-25.1.0 linkify-it-py-2.0.3 markdown-3.6 markdown-it-py-3.0.0 matplotlib-inline-0.1.7 mdit-py-plugins-0.4.0 mdurl-0.1.2 mistune-3.0.2 more-itertools-10.2.0 nbclient-0.10.0 nbconvert-7.16.3 nbformat-5.10.4 nebari-jupyterhub-theme-2023.4.1 nest-asyncio-1.6.0 notebook-7.1.3 notebook-shim-0.2.4 numpy-1.26.4 overrides-7.7.0 pandas-2.2.2 pandocfilters-1.5.1 panel-1.4.1 param-2.1.0 parso-0.8.4 pathspec-0.12.1 pexpect-4.9.0 pillow-10.3.0 platformdirs-4.2.0 plotlydash-tornado-cmd-0.0.6 pluggy-1.5.0 prompt-toolkit-3.0.43 ptyprocess-0.7.0 pure-eval-0.2.2 pydantic-2.7.0 pydantic-core-2.18.1 pygments-2.17.2 python-jose-3.3.0 python-keycloak-0.26.1 python-multipart-0.0.9 pytz-2024.1 pyviz-comms-3.0.2 pyzmq-26.0.2 qtconsole-5.5.1 qtpy-2.4.1 requests-2.31.0 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rich-13.7.1 send2trash-1.8.3 shellingham-1.5.4 simpervisor-0.4 sniffio-1.3.1 soupsieve-2.5 stack-data-0.6.3 starlette-0.37.2 structlog-24.1.0 terminado-0.18.1 tinycss2-1.2.1 tomli-2.0.1 tomli-w-1.0.0 tomlkit-0.12.4 trove-classifiers-2024.4.10 types-python-dateutil-2.9.0.20240316 tzdata-2024.1 uc-micro-py-1.0.3 uri-template-1.3.0 userpath-1.9.2 uvicorn-0.29.0 virtualenv-20.25.3 wcwidth-0.2.13 webcolors-1.13 webencodings-0.5.1 widgetsnbextension-4.0.10 xyzservices-2024.4.0 zstandard-0.22.0

krassowski commented 6 months ago

My current theory is that it has something with helm chart setup as per https://github.com/nebari-dev/nebari/issues/2330#issuecomment-2059001313 but there still isn't a version we could upgrade too. I sent a message on gitter to ask will follow up here.

aktech commented 6 months ago

My current theory is that it has something with helm chart setup as per https://github.com/nebari-dev/nebari/issues/2330#issuecomment-2059001313 but there still isn't a version we could upgrade too. I sent a message on gitter to ask will follow up here.

Ah, yes that's 100% correct. I totally forgot about helm chart. We can fork that repo and create a chart if it's going to take a while to release.

krassowski commented 6 months ago

Things are looking good with with JupyterHub Helm 4.0.0-0.dev.git.6586.h0a16e5a0:

image

There will be some only minor changes needed to address https://github.com/jupyterhub/jupyterhub/issues/4801 - I opened a draft PR https://github.com/nebari-dev/nebari/pull/2427.

krassowski commented 6 months ago

Just as we merged JupyterHub 5.0.0b1 yesterday evening, 5.0.0b2 was released today morning. The only noteworthy change is transition from jupyter-telemetry to jupyter-events. I think this should not cause any issues, but it may be worth bumping the versions to test 5.0.0b2 already.

aktech commented 6 months ago

Good idea.