Upgrade to a newer Python version documentation not functional for python3.10.x

bigpick commented 2 years ago

Bug description

Following the instructions in the documentation on how to Upgrade to a newer Python version, no users are able to create a new server, admin or not.

The default TLJH setup works fine, and I've been able to run it as is on a manual setup w/HTTPs+etc. just fine. However, attempting to upgrade the python version for users to 3.10.x, the server functionality is entirely broken.

Expected behaviour

Following the steps in the documentation should result in the desired state (upgraded python version) with a still functional hub setup.

Actual behaviour

The hub fails to create any new user server's after the doc steps have been following (slightly modified so they actually work, bc as they're written now are also broken; the ... @ file:///... conda packages all get removed and are thus broken when the last pip install ... is attempted. I've tried simply removing those packages entirely, just removing the @ ... and also doing pip list --format=freeze and using that package, none of which seem to have any effect on the problem here, which is the hub being unable to create any new user servers after upgrade.

journalctl logs (see below logs section) for the user's process seems to show it complaining about a jupyterhub package missing, but if I manually check the script that it say's is being invoked, it seems to show that is installed, and that the python at the script's shebang can even import it itself?:

cat /opt/tljh/hub/bin/jupyterhub-singleuser
#!/opt/tljh/hub/bin/python
...

# executing interpreter using above /opt/tljh/hub/bin/python
Python 3.10.4 (main, Jun 29 2022, 12:14:53) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import jupyterhub
>>>

and

/opt/tljh/user/bin/python -m pip list | grep jupy
jupyter-client         7.3.4
jupyter-core           4.11.1
jupyter-resource-usage 0.6.1
jupyter-server         1.18.1
jupyter-telemetry      0.1.0
jupyterhub             1.5.0
jupyterlab             3.4.3
jupyterlab-pygments    0.2.2
jupyterlab-server      2.15.0
jupyterlab-widgets     1.1.1
nteract-on-jupyter     2.1.3

How to reproduce

Install TLJH on a fresh server installation
Manually access the server running TLJH, and follow the steps to upgrade to a newer python version:

# been doing this as a non root user? the docs don't indicate either way; presumably is fine since latter steps use sudo?
source /opt/tljh/user/bin/activate

pip freeze > pip_pkgs.txt # this needs to be fixed/changed too, but isn't this problem -- also, this shouldn't be root?

sudo env PATH=${PATH} conda update --all # this also needed to be fixed, should be `sudo env PATH=...` otherwise it still can't find conda?

sudo env PATH=${PATH} conda install python=3.10 # same env problem as above

pip install -r pip_pkgs.txt # this will fail unless you've removed the conda packages/used the pip list --format=freeze command to generate
# this pip install seems to put everything into the user's `~/.local/bin` and complains about PATH?

Access the Hub, and sign in; first time admin login for example
Go to hub homepage or click the button to create your user's server
Wait for create to fail and timeout (this fails for the first user ie initial admin login, but I also created additional non-admin users and tested their server's too and they failed to start for the same reason)

Your personal set up

OS:
- Distributor ID: Ubuntu
- Description: Ubuntu 22.04 LTS
- Release: 22.04
- Codename: jammy
Version(s):

Before upgrade:

conda --version
conda 4.10.3

[/opt/tljh/hub/bin/python, /opt/tljh/hub/bin/python3, /opt/tljh/hub/bin/python3.10] --version
Python 3.10.4

[/opt/tljh/user/bin/python, /opt/tljh/user/bin/python3, /opt/tljh/user/bin/python3.9] --version
Python 3.9.7

/opt/tljh/hub/bin/jupyterhub --version 
1.5.0

/opt/tljh/user/condabin/conda --version
conda 4.10.3

/opt/tljh/user/bin/jupyterhub --version
1.5.0

After upgrade:

conda --version
conda 4.13.0

[/opt/tljh/hub/bin/python, /opt/tljh/hub/bin/python3, /opt/tljh/hub/bin/python3.10] --version
Python 3.10.4

[/opt/tljh/user/bin/python, /opt/tljh/user/bin/python3, /opt/tljh/user/bin/python3.10] --version
Python 3.10.5

/opt/tljh/hub/bin/jupyterhub --version 
1.5.0

/opt/tljh/user/condabin/conda --version
conda 4.13.0

/opt/tljh/user/bin/jupyterhub --version
1.5.0

Full environment

``` # from inside the conda venv, BEFORE upgrade alembic==1.8.1 anyio==3.6.1 argon2-cffi==21.3.0 argon2-cffi-bindings==21.2.0 asttokens==2.0.5 async-generator==1.10 attrs==21.4.0 Babel==2.10.3 backcall==0.2.0 beautifulsoup4==4.11.1 bleach==5.0.1 brotlipy==0.7.0 certifi==2022.6.15 certipy==0.1.3 cffi==1.14.6 chardet==4.0.0 charset-normalizer==2.0.0 colorama==0.4.4 conda==4.10.3 conda-package-handling==1.7.3 cryptography==3.4.8 debugpy==1.6.2 decorator==5.1.1 defusedxml==0.7.1 entrypoints==0.4 executing==0.8.3 fastjsonschema==2.16.1 greenlet==1.1.2 idna==3.1 importlib-metadata==4.12.0 ipykernel==6.15.1 ipython==8.4.0 ipython-genutils==0.2.0 ipywidgets==7.7.1 jedi==0.18.1 Jinja2==3.1.2 json5==0.9.8 jsonschema==4.7.2 jupyter-client==7.3.4 jupyter-core==4.11.1 jupyter-resource-usage==0.6.1 jupyter-server==1.18.1 jupyter-telemetry==0.1.0 jupyterhub==1.5.0 jupyterlab==3.4.3 jupyterlab-pygments==0.2.2 jupyterlab-server==2.15.0 jupyterlab-widgets==1.1.1 Mako==1.2.1 mamba==0.16.0 MarkupSafe==2.1.1 matplotlib-inline==0.1.3 mistune==0.8.4 nbclassic==0.4.3 nbclient==0.6.6 nbconvert==6.5.0 nbformat==5.4.0 nbgitpuller==1.1.0 nest-asyncio==1.5.5 notebook==6.4.12 notebook-shim==0.1.0 nteract-on-jupyter==2.1.3 oauthlib==3.2.0 packaging==21.3 pamela==1.0.0 pandocfilters==1.5.0 parso==0.8.3 pexpect==4.8.0 pickleshare==0.7.5 pip==21.3 prometheus-client==0.14.1 prompt-toolkit==3.0.30 psutil==5.9.1 ptyprocess==0.7.0 pure-eval==0.2.2 pycosat==0.6.3 pycparser==2.20 Pygments==2.12.0 pyOpenSSL==21.0.0 pyparsing==3.0.9 pyrsistent==0.18.1 PySocks==1.7.1 python-dateutil==2.8.2 python-json-logger==2.0.4 pytz==2022.1 pyzmq==23.2.0 requests==2.26.0 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.6 ruamel-yaml-conda==0.15.80 Send2Trash==1.8.0 setuptools==58.2.0 six==1.16.0 sniffio==1.2.0 soupsieve==2.3.2.post1 SQLAlchemy==1.4.39 stack-data==0.3.0 terminado==0.15.0 tinycss2==1.1.1 tornado==6.2 tqdm==4.62.3 traitlets==5.3.0 urllib3==1.26.7 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.3 wheel==0.37.0 widgetsnbextension==3.6.1 zipp==3.8.1 ``` from inside venv, AFTER upgrade ``` alembic==1.8.1 anyio==3.6.1 argon2-cffi==21.3.0 argon2-cffi-bindings==21.2.0 asttokens==2.0.5 async-generator==1.10 attrs==21.4.0 Babel==2.10.3 backcall==0.2.0 beautifulsoup4==4.11.1 bleach==5.0.1 brotlipy==0.7.0 certifi==2022.6.15 certipy==0.1.3 cffi==1.15.1 chardet==4.0.0 charset-normalizer==2.1.0 colorama==0.4.5 conda==4.13.0 conda-package-handling==1.8.1 cryptography==37.0.4 debugpy==1.6.2 decorator==5.1.1 defusedxml==0.7.1 entrypoints==0.4 executing==0.8.3 fastjsonschema==2.16.1 greenlet==1.1.2 idna==3.3 importlib-metadata==4.12.0 ipykernel==6.15.1 ipython==8.4.0 ipython-genutils==0.2.0 ipywidgets==7.7.1 jedi==0.18.1 Jinja2==3.1.2 json5==0.9.8 jsonschema==4.7.2 jupyter-client==7.3.4 jupyter-core==4.11.1 jupyter-resource-usage==0.6.1 jupyter-server==1.18.1 jupyter-telemetry==0.1.0 jupyterhub==1.5.0 jupyterlab==3.4.3 jupyterlab-pygments==0.2.2 jupyterlab-server==2.15.0 jupyterlab-widgets==1.1.1 libmambapy==0.24.0 Mako==1.2.1 mamba==0.24.0 MarkupSafe==2.1.1 matplotlib-inline==0.1.3 mistune==0.8.4 nbclassic==0.4.3 nbclient==0.6.6 nbconvert==6.5.0 nbformat==5.4.0 nbgitpuller==1.1.0 nest-asyncio==1.5.5 notebook==6.4.12 notebook-shim==0.1.0 nteract-on-jupyter==2.1.3 oauthlib==3.2.0 packaging==21.3 pamela==1.0.0 pandocfilters==1.5.0 parso==0.8.3 pexpect==4.8.0 pickleshare==0.7.5 pip==21.3 prometheus-client==0.14.1 prompt-toolkit==3.0.30 psutil==5.9.1 ptyprocess==0.7.0 pure-eval==0.2.2 pycosat==0.6.3 pycparser==2.21 Pygments==2.12.0 pyOpenSSL==22.0.0 pyparsing==3.0.9 pyrsistent==0.18.1 PySocks==1.7.1 python-dateutil==2.8.2 python-json-logger==2.0.4 pytz==2022.1 pyzmq==23.2.0 requests==2.28.1 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.6 ruamel-yaml-conda==0.15.80 Send2Trash==1.8.0 setuptools==58.2.0 six==1.16.0 sniffio==1.2.0 soupsieve==2.3.2.post1 SQLAlchemy==1.4.39 stack-data==0.3.0 terminado==0.15.0 tinycss2==1.1.1 tornado==6.2 tqdm==4.64.0 traitlets==5.3.0 urllib3==1.26.10 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.3 wheel==0.37.0 widgetsnbextension==3.6.1 zipp==3.8.1 ```

Configuration

```shell # from inside the venv, BEFORE upgrade: grep -v '$^#\|^[[:space:]]*$$' jupyterhub_config.py ```

Logs

For `journalctl -u jupyterhub` when trying to start a user's server AFTER upgrade: ``` Jul 18 22:32:51 test-tljh-server python3[6577]: [I 2022-07-18 22:32:51.883 JupyterHub log:189] 302 GET / -> /hub/ (@127.0.0.1) 1.13ms Jul 18 22:32:52 test-tljh-server python3[6577]: [W 2022-07-18 22:32:52.024 JupyterHub base:390] Invalid or expired cookie token Jul 18 22:32:52 test-tljh-server python3[6577]: [I 2022-07-18 22:32:52.025 JupyterHub log:189] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@127.0.0.1) 1.51ms Jul 18 22:32:52 test-tljh-server python3[6577]: [I 2022-07-18 22:32:52.115 JupyterHub log:189] 200 GET /hub/login?next=%2Fhub%2F (@127.0.0.1) 24.61ms Jul 18 22:32:58 test-tljh-server python3[6577]: [I 2022-07-18 22:32:58.718 JupyterHub base:762] User logged in: testoperator Jul 18 22:32:58 test-tljh-server python3[6577]: [I 2022-07-18 22:32:58.718 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (testoperator@127.0.0.1) 246.62ms Jul 18 22:32:58 test-tljh-server python3[6577]: [I 2022-07-18 22:32:58.909 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn (testoperator@127.0.0.1) 42.63ms Jul 18 22:32:59 test-tljh-server python3[6577]: [I 2022-07-18 22:32:59.001 JupyterHub provider:574] Creating oauth client jupyterhub-user-testoperator Jul 18 22:32:59 test-tljh-server useradd[8684]: new group: name=jupyter-testoperator, GID=1004 Jul 18 22:32:59 test-tljh-server useradd[8684]: new user: name=jupyter-testoperator, UID=1002, GID=1004, home=/home/jupyter-testoperator, shell=/bin/sh, from=none Jul 18 22:32:59 test-tljh-server python3[8693]: Adding user jupyter-testoperator to group jupyterhub-users Jul 18 22:32:59 test-tljh-server gpasswd[8693]: user jupyter-testoperator added by root to group jupyterhub-users Jul 18 22:32:59 test-tljh-server python3[8699]: Adding user jupyter-testoperator to group jupyterhub-admins Jul 18 22:32:59 test-tljh-server gpasswd[8699]: user jupyter-testoperator added by root to group jupyterhub-admins Jul 18 22:32:59 test-tljh-server python3[8709]: Running as unit: jupyter-testoperator.service Jul 18 22:32:59 test-tljh-server python3[6577]: [I 2022-07-18 22:32:59.991 JupyterHub log:189] 302 GET /hub/spawn -> /hub/spawn-pending/testoperator (testoperator@127.0.0.1) 1005.75ms Jul 18 22:33:00 test-tljh-server python3[6577]: [I 2022-07-18 22:33:00.098 JupyterHub pages:402] testoperator is pending spawn Jul 18 22:33:00 test-tljh-server python3[6577]: [I 2022-07-18 22:33:00.102 JupyterHub log:189] 200 GET /hub/spawn-pending/testoperator (testoperator@127.0.0.1) 5.87ms Jul 18 22:33:08 test-tljh-server python3[6577]: Task exception was never retrieved Jul 18 22:33:08 test-tljh-server python3[6577]: future: exception=HTTPError()> Jul 18 22:33:08 test-tljh-server python3[6577]: Traceback (most recent call last): Jul 18 22:33:08 test-tljh-server python3[6577]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 996, in spawn_single_user Jul 18 22:33:08 test-tljh-server python3[6577]: await gen.with_timeout( Jul 18 22:33:08 test-tljh-server python3[6577]: asyncio.exceptions.TimeoutError: Timeout Jul 18 22:33:08 test-tljh-server python3[6577]: During handling of the above exception, another exception occurred: Jul 18 22:33:08 test-tljh-server python3[6577]: Traceback (most recent call last): Jul 18 22:33:08 test-tljh-server python3[6577]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 1030, in spawn_single_user Jul 18 22:33:08 test-tljh-server python3[6577]: raise web.HTTPError( Jul 18 22:33:08 test-tljh-server python3[6577]: tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for testoperator may contain details.) Jul 18 22:33:19 test-tljh-server python3[6577]: [I 2022-07-18 22:33:19.178 JupyterHub log:189] 200 GET /hub/api/ (cull-idle@127.0.0.1) 8.13ms Jul 18 22:33:19 test-tljh-server python3[6577]: [I 2022-07-18 22:33:19.182 JupyterHub log:189] 200 GET /hub/api/users?state=[secret] (cull-idle@127.0.0.1) 2.71ms Jul 18 22:33:31 test-tljh-server python3[6577]: [W 2022-07-18 22:33:31.817 JupyterHub user:767] testoperator's server never showed up at http://127.0.0.1:46559/user/testoperator/ after 30 seconds. Giving up Jul 18 22:33:31 test-tljh-server python3[6577]: [E 2022-07-18 22:33:31.844 JupyterHub gen:630] Exception in Future .finish_user_spawn() done, defined at /opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py:900> exception=TimeoutError("Server at http://127.0.0.1:46559/user/testoperator/ didn't respond in 30 seconds")> after timeout Jul 18 22:33:31 test-tljh-server python3[6577]: Traceback (most recent call last): Jul 18 22:33:31 test-tljh-server python3[6577]: File "/opt/tljh/hub/lib/python3.10/site-packages/tornado/gen.py", line 625, in error_callback Jul 18 22:33:31 test-tljh-server python3[6577]: future.result() Jul 18 22:33:31 test-tljh-server python3[6577]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn Jul 18 22:33:31 test-tljh-server python3[6577]: await spawn_future Jul 18 22:33:31 test-tljh-server python3[6577]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/user.py", line 748, in spawn Jul 18 22:33:31 test-tljh-server python3[6577]: await self._wait_up(spawner) Jul 18 22:33:31 test-tljh-server python3[6577]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/user.py", line 795, in _wait_up Jul 18 22:33:31 test-tljh-server python3[6577]: raise e Jul 18 22:33:31 test-tljh-server python3[6577]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/user.py", line 762, in _wait_up Jul 18 22:33:31 test-tljh-server python3[6577]: resp = await server.wait_up( Jul 18 22:33:31 test-tljh-server python3[6577]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/utils.py", line 236, in wait_for_http_server Jul 18 22:33:31 test-tljh-server python3[6577]: re = await exponential_backoff( Jul 18 22:33:31 test-tljh-server python3[6577]: File "/opt/tljh/hub/lib/python3.10/site-packages/jupyterhub/utils.py", line 184, in exponential_backoff Jul 18 22:33:31 test-tljh-server python3[6577]: raise TimeoutError(fail_message) Jul 18 22:33:31 test-tljh-server python3[6577]: TimeoutError: Server at http://127.0.0.1:46559/user/testoperator/ didn't respond in 30 seconds Jul 18 22:33:31 test-tljh-server python3[6577]: Jul 18 22:33:31 test-tljh-server python3[6577]: [I 2022-07-18 22:33:31.846 JupyterHub log:189] 200 GET /hub/api/users/testoperator/server/progress (testoperator@127.0.0.1) 31593.27ms ``` For `journalctl -u testoperator.service`: ``` Jul 18 22:32:59 test-tljh-server systemd[1]: Started /bin/bash -c cd /home/jupyter-testoperator && exec jupyterhub-singleuser --port=46559. Jul 18 22:32:59 test-tljh-server bash[8711]: Traceback (most recent call last): Jul 18 22:32:59 test-tljh-server bash[8711]: File "/opt/tljh/user/bin/jupyterhub-singleuser", line 5, in Jul 18 22:32:59 test-tljh-server bash[8711]: from jupyterhub.singleuser import main Jul 18 22:32:59 test-tljh-server bash[8711]: ModuleNotFoundError: No module named 'jupyterhub' Jul 18 22:32:59 test-tljh-server systemd[1]: jupyter-testoperator.service: Main process exited, code=exited, status=1/FAILURE Jul 18 22:32:59 test-tljh-server systemd[1]: jupyter-testoperator.service: Failed with result 'exit-code'. ```

welcome[bot] commented 2 years ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

bigpick commented 2 years ago

In trying a bunch of different stuff, it seems like maybe just doing a

sudo env "PATH=$PATH" pip install <reqs_without_conda>

at the end (vs. just the documented pip install -r pip_pkgs.txt) works? Going to scrap and try again from scratch with just that and see

bigpick commented 2 years ago

Doing the following seems to have worked now; Python versions show 3.10.x (3.10.5 in this case) in the terminal from the hub as a user, new user servers are able to be spun up again; haven't seen any issues in the minor manual testing I've done so far:

source /opt/tljh/user/bin/activate

conda env export --json | jq -r '.dependencies[] | objects | .pip[]' > pip_reqs_from_conda.txt

sudo env PATH=${PATH} conda update --all  # has confirmation prompt

sudo env PATH=${PATH} conda install python=3.10  # has confirmation prompt

sudo env "PATH=$PATH" pip install -r pip_reqs_from_conda.txt

Will open a PR in a bit

mtav commented 2 years ago

Your solution fails for me at the python 3.10 installation step with the following output:

$ sudo env PATH=${PATH} conda install python=3.10  # has confirmation prompt
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /opt/tljh/user

  added / updated specs:
    - python=3.10

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    brotlipy-0.7.0             |py310h5764c6d_1004         342 KB  conda-forge
    cffi-1.15.1                |  py310h255011f_0         231 KB  conda-forge
    conda-4.14.0               |  py310hff52083_0         1.0 MB  conda-forge
    conda-package-handling-1.8.1|  py310h5764c6d_1        1015 KB  conda-forge
    cryptography-37.0.4        |  py310h597c629_0         1.6 MB  conda-forge
    libmambapy-0.25.0          |  py310hab0e683_2         297 KB  conda-forge
    libnsl-2.0.0               |       h7f98852_0          31 KB  conda-forge
    libsqlite-3.39.3           |       h753d276_0         789 KB  conda-forge
    libuuid-2.32.1             |    h7f98852_1000          28 KB  conda-forge
    mamba-0.25.0               |  py310hf87f941_2          46 KB  conda-forge
    ncurses-6.3                |       h27087fc_1        1002 KB  conda-forge
    pycosat-0.6.3              |py310h5764c6d_1010         108 KB  conda-forge
    python-3.10.6              |h582c2e5_0_cpython        29.0 MB  conda-forge
    python_abi-3.10            |          2_cp310           4 KB  conda-forge
    readline-8.1.2             |       h0f457ee_0         291 KB  conda-forge
    ruamel_yaml-0.15.80        |py310h5764c6d_1007         249 KB  conda-forge
    ------------------------------------------------------------
                                           Total:        35.8 MB

The following NEW packages will be INSTALLED:

  libnsl             conda-forge/linux-64::libnsl-2.0.0-h7f98852_0
  libsqlite          conda-forge/linux-64::libsqlite-3.39.3-h753d276_0
  libuuid            conda-forge/linux-64::libuuid-2.32.1-h7f98852_1000

The following packages will be REMOVED:

  sqlite-3.36.0-h9cd32fc_2

The following packages will be UPDATED:

  ncurses                                    6.2-h58526e2_4 --> 6.3-h27087fc_1
  python                           3.9.7-hb7a2778_3_cpython --> 3.10.6-h582c2e5_0_cpython
  python_abi                                     3.9-2_cp39 --> 3.10-2_cp310
  readline                                   8.1-h46c0cb4_0 --> 8.1.2-h0f457ee_0

The following packages will be DOWNGRADED:

  brotlipy                          0.7.0-py39hb9d737c_1004 --> 0.7.0-py310h5764c6d_1004
  cffi                                1.15.1-py39he91dace_0 --> 1.15.1-py310h255011f_0
  conda                               4.14.0-py39hf3d152e_0 --> 4.14.0-py310hff52083_0
  conda-package-han~                   1.8.1-py39hb9d737c_1 --> 1.8.1-py310h5764c6d_1
  cryptography                        37.0.4-py39hd97740a_0 --> 37.0.4-py310h597c629_0
  libmambapy                          0.25.0-py39hd55135b_2 --> 0.25.0-py310hab0e683_2
  mamba                               0.25.0-py39hfa8f2c8_2 --> 0.25.0-py310hf87f941_2
  pycosat                           0.6.3-py39hb9d737c_1010 --> 0.6.3-py310h5764c6d_1010
  ruamel_yaml                     0.15.80-py39hb9d737c_1007 --> 0.15.80-py310h5764c6d_1007

Proceed ([y]/n)? y

Downloading and Extracting Packages
conda-package-handli | 1015 KB   | ################################################################################################################################################################################################ | 100% 
brotlipy-0.7.0       | 342 KB    | ################################################################################################################################################################################################ | 100% 
cryptography-37.0.4  | 1.6 MB    | ################################################################################################################################################################################################ | 100% 
python-3.10.6        | 29.0 MB   | ################################################################################################################################################################################################ | 100% 
cffi-1.15.1          | 231 KB    | ################################################################################################################################################################################################ | 100% 
pycosat-0.6.3        | 108 KB    | ################################################################################################################################################################################################ | 100% 
libuuid-2.32.1       | 28 KB     | ################################################################################################################################################################################################ | 100% 
python_abi-3.10      | 4 KB      | ################################################################################################################################################################################################ | 100% 
mamba-0.25.0         | 46 KB     | ################################################################################################################################################################################################ | 100% 
libmambapy-0.25.0    | 297 KB    | ################################################################################################################################################################################################ | 100% 
conda-4.14.0         | 1.0 MB    | ################################################################################################################################################################################################ | 100% 
ncurses-6.3          | 1002 KB   | ################################################################################################################################################################################################ | 100% 
readline-8.1.2       | 291 KB    | ################################################################################################################################################################################################ | 100% 
ruamel_yaml-0.15.80  | 249 KB    | ################################################################################################################################################################################################ | 100% 
libnsl-2.0.0         | 31 KB     | ################################################################################################################################################################################################ | 100% 
libsqlite-3.39.3     | 789 KB    | ################################################################################################################################################################################################ | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Retrieving notices: ...working... failed
Traceback (most recent call last):
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1125, in __call__
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/cli/main.py", line 86, in main_subshell
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 93, in do_call
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/core.py", line 75, in wrapper
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/core.py", line 39, in display_notices
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/http.py", line 36, in get_notice_responses
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/http.py", line 39, in <genexpr>
  File "/opt/tljh/user/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
  File "/opt/tljh/user/lib/python3.9/concurrent/futures/_base.py", line 445, in result
  File "/opt/tljh/user/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
  File "/opt/tljh/user/lib/python3.9/concurrent/futures/thread.py", line 52, in run
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/http.py", line 42, in <lambda>
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/cache.py", line 37, in wrapper
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/notices/http.py", line 58, in get_channel_notice_response
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/sessions.py", line 701, in send
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/adapters.py", line 460, in send
  File "/opt/tljh/user/lib/python3.9/site-packages/requests/adapters.py", line 263, in cert_verify
OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /opt/tljh/user/lib/python3.9/site-packages/certifi/cacert.pem

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/tljh/user/bin/conda", line 13, in <module>
    sys.exit(main())
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/cli/main.py", line 129, in main
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1413, in conda_exception_handler
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1128, in __call__
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1170, in handle_exception
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1181, in handle_unexpected_exception
  File "/opt/tljh/user/lib/python3.9/site-packages/conda/exceptions.py", line 1251, in print_unexpected_error_report
ModuleNotFoundError: No module named 'conda.cli.main_info'

eidalex commented 2 years ago

Same problem here with a JupterHub installed on an Azure VM with Linux Server 22.04 LTS. Once updating to python 3.10, I had the same message "no module named 'conda.cli.main_info'" and now it is impossible to spawn servers again.

mtav commented 2 years ago

I tried figuring out why it installs python 3.9 in the user environment and got as far as this part of the installer code: https://github.com/jupyterhub/the-littlest-jupyterhub/blob/main/tljh/installer.py#L156

Maybe setting different versions in there would lead to python 3.10. But I assume it still uses python 3.9 for stability reasons?

In the end, I decided to work around the problem by simply creating a new conda/jupyter environment with python 3.10. The following code should work after logging into the server via ssh:

# activate environment
source /opt/tljh/user/bin/activate
# create new environment
sudo env PATH=${PATH} conda create -n "py3.10" python=3.10
# activate it
conda activate "py3.10"
# install ipykernel
sudo env PATH=${PATH} CONDA_DEFAULT_ENV="py3.10" conda install ipykernel
# make available in jupyter
sudo env PATH=${PATH} CONDA_DEFAULT_ENV="py3.10" ipython kernel install --name="py3.10"

# check it got added to conda
conda env list
# check it got added to jupyter
jupyter kernelspec list

I suspect that the CONDA_DEFAULT_ENV="py3.10" parts are not necessary when using a terminal launched from a jupyterhub as admin, but haven't extensively tested it.

Running conda init bash after ssh-ing into the server and relogging in might also help, but I prefer not to mess up my .bashrc on the server.

mtav commented 2 years ago

I just discovered there is a pull request for allowing specifying the python version on install: #784 Not quite a solution to the upgrade problem, but it will certainly help people doing a fresh install, who want a specific python version.

SamuelAl commented 1 year ago

None of these methods are working for me. Is there no straightforward way to update the user environment to Python 3.10? Is this a compatibility issue?

kiwifb commented 1 year ago

I am currently testing some code allowing you to use a different version of mambaforge than the one hard-coded in https://github.com/jupyterhub/the-littlest-jupyterhub/blob/main/tljh/installer.py#L156. This enabled me to get python 3.10 out of the box by pulling the latest mambaforge. So far, I have no complaints from my users. The code is at https://github.com/kiwifb/the-littlest-jupyterhub/tree/mambaforge_env

jupyterhub / the-littlest-jupyterhub