search

jupyterhub / the-littlest-jupyterhub

Simple JupyterHub distribution for 1-100 users on a single server
https://tljh.jupyter.org
BSD 3-Clause "New" or "Revised" License
1.05k stars 343 forks source link

Spawner not running correct python engine TLJH #809

Open hyamanieu opened 2 years ago

hyamanieu commented 2 years ago

Note: I have set up "the littlest jupyter hub" on Ubuntu 18.04 a few years ago. Everything went well until last week.

Bug description

Using a administrator account, I have update jupyterlab in the base conda. From what I understand, it corresponds to a conda environment under /opt/tljh/user.

Everything looked OK until monday when starting a user server would not "spawn" (time out 30 seconds). I'll copy what I get in journalctl:

Apr 12 09:35:08 jupyterhub-prod systemd[1]: Started /bin/bash -c cd /home/jupyter-hy.amanieu && exec jupyterhub-singleuser --port=58621 --SingleUserNotebookApp.default_url=/lab.
Apr 12 09:35:08 jupyterhub-prod bash[29250]: Traceback (most recent call last):
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/bin/jupyterhub-singleuser", line 5, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from jupyterhub.singleuser import main
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser/__init__.py", line 5, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .app import main
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser/app.py", line 10, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .mixins import make_singleuser_app
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser/mixins.py", line 45, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from ..log import log_request
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/log.py", line 17, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .handlers.pages import HealthCheckHandler
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/handlers/__init__.py", line 1, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from . import base
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 32, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .. import orm
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/orm.py", line 42, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .utils import compare_token
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/utils.py", line 24, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from async_generator import aclosing
Apr 12 09:35:08 jupyterhub-prod bash[29250]: ModuleNotFoundError: No module named 'async_generator'
Apr 12 09:35:08 jupyterhub-prod systemd[1]: jupyter-hy.amanieu.service: Main process exited, code=exited, status=1/FAILURE
Apr 12 09:35:08 jupyterhub-prod systemd[1]: jupyter-hy.amanieu.service: Failed with result 'exit-code'.

As you can see, async-generator is missing. Well it is of course not missing from any of my python environment: the hub under /opt/tljh/hub or the main under /opt/tljh/user. I have directly tested the python under the ./bin/ subfolders: all is there.

Typically I would do: <path>/python3 -m pip list | grep async and it would show.

I have checked what is inside the file which starts the spawner: /opt/tljh/user/bin/jupyterhub-singleuser. I could see the following she-bang: #!/opt/tljh/user/bin/python . That engine can import async-generator.

I have no idea how to add more information in the systemd journal for the user to understand which python engine is spawning it and from which packages it's trying to load it.

Expected behaviour

The user server should spawn

Actual behaviour

it doesn't spawn

How to reproduce

I have no idea how to reproduce it. I cannot afford deleting everything to try to reproduce it, this server is important.

Your personal set up

Full environment of opt/tljh/user ``` WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(144): Could not remove or rename /opt/tljh/user/conda-meta/widgetsnbextension-3.5.1-pyh9f0ad1d_3.json. Please remove this file manually (you may need to reboot to free file handles) WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(144): Could not remove or rename /opt/tljh/user/conda-meta/ipywidgets-7.6.3-pyhd3deb0d_0.json. Please remove this file manually (you may need to reboot to free file handles) # packages in environment at /opt/tljh/user: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge aiohttp 3.8.1 py37h540881e_1 conda-forge aiosignal 1.2.0 pyhd8ed1ab_0 conda-forge alembic 1.4.2 pypi_0 pypi anyio 3.5.0 py37h89c1867_0 conda-forge argcomplete 2.0.0 pyhd8ed1ab_0 conda-forge argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py37h5e8e339_1 conda-forge async-timeout 4.0.2 pyhd8ed1ab_0 conda-forge asynctest 0.13.0 py_0 conda-forge attrs 21.4.0 pyhd8ed1ab_0 conda-forge babel 2.9.1 pyh44b312d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.11.0 pyha770c72_0 conda-forge bleach 3.1.4 pypi_0 pypi bokeh 2.4.2 py37h89c1867_0 conda-forge bottleneck 1.3.4 py37h6c7ee08_0 conda-forge bqplot 0.12.33 pyhd8ed1ab_0 conda-forge brotli 1.0.9 h166bdaf_7 conda-forge brotli-bin 1.0.9 h166bdaf_7 conda-forge brotli-python 1.0.9 py37hd23a5d3_7 conda-forge brotlipy 0.7.0 py37h540881e_1004 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge ca-certificates 2022.3.29 h06a4308_0 certifi 2021.10.8 py37h89c1867_2 conda-forge certipy 0.1.3 pypi_0 pypi cffi 1.15.0 py37h7f8727e_0 charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge click 8.1.2 py37h89c1867_0 conda-forge colorama 0.4.4 pyh9f0ad1d_0 conda-forge conda 4.10.3 py37h89c1867_4 conda-forge conda-package-handling 1.8.1 py37h540881e_1 conda-forge cryptography 36.0.2 py37h38fbfac_1 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge dash 2.3.1 pyhd8ed1ab_0 conda-forge dash-bootstrap-components 1.1.0 pyhd8ed1ab_0 conda-forge dash-daq 0.5.0 pyh9f0ad1d_1 conda-forge dash_colorscales 0.0.4 pyh9f0ad1d_0 conda-forge dbus 1.13.6 he372182_0 conda-forge debugpy 1.5.1 py37hcd2ae1e_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge dtale 1.55.0 pyhd8ed1ab_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge et_xmlfile 1.0.1 py_1001 conda-forge expat 2.4.8 h27087fc_0 conda-forge flask 2.1.1 pyhd8ed1ab_0 conda-forge flask-compress 1.11 pyhd8ed1ab_0 conda-forge flit-core 3.7.1 pyhd8ed1ab_0 conda-forge fontconfig 2.14.0 h8e229c2_0 conda-forge fonttools 4.31.2 py37h540881e_1 conda-forge freetype 2.10.4 h0708190_1 conda-forge frozenlist 1.3.0 py37h540881e_1 conda-forge future 0.18.2 py37h89c1867_5 conda-forge gast 0.4.0 pyh9f0ad1d_0 conda-forge gettext 0.19.8.1 hf34092f_1004 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge glib 2.58.3 py37he00f558_1004 conda-forge gst-plugins-base 1.14.5 h0935bb2_2 conda-forge gstreamer 1.14.5 h36ae1b5_2 conda-forge icu 58.2 hf484d3e_1000 conda-forge idna 3.3 pyhd8ed1ab_0 conda-forge importlib-metadata 1.6.0 pypi_0 pypi importlib_metadata 4.11.3 hd8ed1ab_1 conda-forge importlib_resources 5.6.0 pyhd8ed1ab_0 conda-forge ipydatagrid 1.1.2 pyhd8ed1ab_0 conda-forge ipykernel 5.2.1 pypi_0 pypi ipyparallel 6.3.0 py37h06a4308_0 ipython 7.13.0 pypi_0 pypi ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.5.1 pypi_0 pypi itsdangerous 2.1.2 pyhd8ed1ab_0 conda-forge jbig 2.1 h7f98852_2003 conda-forge jedi 0.18.1 py37h89c1867_1 conda-forge jinja2 3.1.1 pyhd8ed1ab_0 conda-forge joblib 1.1.0 pyhd8ed1ab_0 conda-forge jpeg 9e h7f98852_0 conda-forge json5 0.9.4 pypi_0 pypi jsonschema 4.4.0 pyhd8ed1ab_0 conda-forge jupyter-resource-usage 0.6.1 pypi_0 pypi jupyter-server-proxy 3.1.0 pyhd8ed1ab_0 conda-forge jupyter_bokeh 3.0.2 pyhd8ed1ab_0 conda-forge jupyter_client 7.2.2 pyhd8ed1ab_1 conda-forge jupyter_core 4.9.2 py37h89c1867_0 conda-forge jupyter_server 1.13.5 pyhd3eb1b0_0 jupyterhub 1.0.0 pypi_0 pypi jupyterlab 1.2.11 pypi_0 pypi jupyterlab-server 1.1.1 pypi_0 pypi jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge jupyterlab_server 2.12.0 pyhd8ed1ab_0 conda-forge jupyterlab_widgets 1.1.0 pyhd8ed1ab_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.2 py37h7cecad7_1 conda-forge krb5 1.19.3 h3790be6_0 conda-forge lcms2 2.12 hddcbb42_0 conda-forge lerc 3.0 h9c3ff4c_0 conda-forge libarchive 3.5.2 hccf745f_1 conda-forge libblas 3.9.0 14_linux64_openblas conda-forge libbrotlicommon 1.0.9 h166bdaf_7 conda-forge libbrotlidec 1.0.9 h166bdaf_7 conda-forge libbrotlienc 1.0.9 h166bdaf_7 conda-forge libcblas 3.9.0 14_linux64_openblas conda-forge libcurl 7.82.0 h7bff187_0 conda-forge libdeflate 1.10 h7f98852_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libffi 3.2.1 he1b5a44_1007 conda-forge libgcc-ng 11.2.0 h1d223b6_15 conda-forge libgfortran-ng 11.2.0 h69a702a_15 conda-forge libgfortran5 11.2.0 h5c6108e_15 conda-forge libgomp 11.2.0 h1d223b6_15 conda-forge libiconv 1.16 h516909a_0 conda-forge liblapack 3.9.0 14_linux64_openblas conda-forge libnghttp2 1.47.0 h727a467_0 conda-forge libopenblas 0.3.20 pthreads_h78a6416_0 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libsolv 0.7.19 h780b84a_5 conda-forge libssh2 1.10.0 ha56f1ee_2 conda-forge libstdcxx-ng 11.2.0 he4da1e4_15 conda-forge libtiff 4.3.0 h542a066_3 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libwebp 1.2.2 h3452ae3_0 conda-forge libwebp-base 1.2.2 h7f98852_1 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxml2 2.9.12 h03d6c58_0 libxslt 1.1.34 hc22bd24_0 libzlib 1.2.11 h166bdaf_1014 conda-forge lxml 4.6.2 py37h9120a33_0 lz4 4.0.0 py37hbaff0ff_1 conda-forge lz4-c 1.9.3 h9c3ff4c_1 conda-forge lzo 2.10 h516909a_1000 conda-forge mako 1.1.2 pypi_0 pypi mamba 0.16.0 py37h7f483ca_0 conda-forge markupsafe 2.1.1 py37h540881e_1 conda-forge matplotlib 3.5.1 py37h89c1867_0 conda-forge matplotlib-base 3.5.1 py37h1058ff1_0 conda-forge matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge missingno 0.4.2 py_1 conda-forge mistune 0.8.4 py37h5e8e339_1005 conda-forge multidict 6.0.2 py37h540881e_1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge nbclassic 0.3.7 pyhd8ed1ab_0 conda-forge nbclient 0.5.13 pyhd8ed1ab_0 conda-forge nbconvert 6.4.5 pyhd8ed1ab_2 conda-forge nbconvert-core 6.4.5 pyhd8ed1ab_2 conda-forge nbconvert-pandoc 6.4.5 pyhd8ed1ab_2 conda-forge nbformat 5.1.3 pyhd8ed1ab_0 conda-forge nbgitpuller 1.1.0 pypi_0 pypi nbresuse 0.3.4 pypi_0 pypi ncurses 6.3 h27087fc_1 conda-forge nest-asyncio 1.5.5 pyhd8ed1ab_0 conda-forge networkx 2.7.1 pyhd8ed1ab_0 conda-forge nodejs 12.4.0 he1b5a44_0 conda-forge nomkl 1.0 h5ca1d4c_0 conda-forge notebook 6.4.3 pyha770c72_0 conda-forge notebook-shim 0.1.0 pyhd8ed1ab_0 conda-forge nteract-on-jupyter 2.1.3 pypi_0 pypi numexpr 2.8.0 py37hfe5f03c_101 conda-forge numpy 1.21.5 py37hf2998dd_0 conda-forge oauthlib 3.1.0 pypi_0 pypi openjpeg 2.4.0 hb52868f_1 conda-forge openpyxl 3.0.9 pyhd8ed1ab_0 conda-forge openssl 1.1.1n h166bdaf_0 conda-forge packaging 21.3 pyhd8ed1ab_0 conda-forge pamela 1.0.0 pypi_0 pypi pandas 1.3.3 py37h8c16a72_0 pandoc 2.17.1.1 ha770c72_0 conda-forge pandocfilters 1.4.2 pypi_0 pypi param 1.12.1 pyh6c4a22f_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge patsy 0.5.2 pyhd8ed1ab_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 9.1.0 py37h44f0d7a_0 conda-forge pip 21.2.2 py37h06a4308_0 plotly 5.7.0 pyhd8ed1ab_0 conda-forge ppscore 1.2.0 pyhd3deb0d_0 conda-forge prometheus-client 0.7.1 pypi_0 pypi prometheus_client 0.14.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.29 pyha770c72_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge py2vega 0.6.1 pyhd8ed1ab_0 conda-forge pycosat 0.6.3 py37h540881e_1010 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.11.2 pyhd8ed1ab_0 conda-forge pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.7 pyhd8ed1ab_0 conda-forge pyqt 5.9.2 py37hcca6a23_4 conda-forge pyrsistent 0.18.1 py37h540881e_1 conda-forge pysocks 1.7.1 py37h89c1867_5 conda-forge python 3.7.3 h0371630_0 python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-editor 1.0.4 pypi_0 pypi python_abi 3.7 2_cp37m conda-forge pytz 2022.1 pyhd8ed1ab_0 conda-forge pyviz_comms 2.1.0 py_0 pyviz pyyaml 6.0 py37h540881e_4 conda-forge pyzmq 19.0.0 pypi_0 pypi qgrid 1.3.1 pyhd8ed1ab_3 conda-forge qt 5.9.7 h52cfd70_2 conda-forge readline 7.0 hf8c457e_1001 conda-forge reproc 14.2.3 h7f98852_0 conda-forge reproc-cpp 14.2.3 h9c3ff4c_0 conda-forge requests 2.27.1 pyhd8ed1ab_0 conda-forge rise 5.7.1 py37h89c1867_0 conda-forge ruamel_yaml 0.15.80 py37h5e8e339_1006 conda-forge scikit-learn 1.0.2 py37hf9e9bfc_0 conda-forge scipy 1.7.3 py37hf2a6cf1_0 conda-forge seaborn 0.11.1 pyhd3eb1b0_0 send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 62.0.0 py37h89c1867_0 conda-forge sharepoint 0.4.2 pypi_0 pypi simpervisor 0.4 pyhd8ed1ab_0 conda-forge sip 4.19.8 py37hf484d3e_0 six 1.16.0 pyh6c4a22f_0 conda-forge sniffio 1.2.0 py37h89c1867_3 conda-forge soupsieve 2.3.1 pyhd8ed1ab_0 conda-forge sqlalchemy 1.3.16 pypi_0 pypi sqlite 3.33.0 h62c20be_0 squarify 0.4.3 py_0 conda-forge statsmodels 0.13.2 py37hb1e94ed_0 conda-forge strsimpy 0.1.9 pyh9f0ad1d_0 conda-forge tenacity 8.0.1 pyhd8ed1ab_0 conda-forge terminado 0.8.3 pypi_0 pypi testpath 0.6.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tk 8.6.12 h27826a3_0 conda-forge tornado 5.1.1 pypi_0 pypi tqdm 4.64.0 pyhd8ed1ab_0 conda-forge traitlets 5.1.1 pyhd8ed1ab_0 conda-forge traittypes 0.2.1 pyh9f0ad1d_2 conda-forge typing-extensions 4.1.1 hd8ed1ab_0 conda-forge typing_extensions 4.1.1 pyha770c72_0 conda-forge unicodedata2 14.0.0 py37h540881e_1 conda-forge urllib3 1.26.9 pyhd8ed1ab_0 conda-forge wcwidth 0.1.9 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.3.2 pyhd8ed1ab_0 conda-forge werkzeug 2.1.1 pyhd8ed1ab_0 conda-forge wheel 0.37.1 pyhd3eb1b0_0 widgetsnbextension 3.6.0 pypi_0 pypi xarray 0.20.2 pyhd8ed1ab_0 conda-forge xlrd 2.0.1 pyhd8ed1ab_3 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h7f98852_2 conda-forge yarl 1.7.2 py37h540881e_2 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zipp 3.8.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h166bdaf_1014 conda-forge zstd 1.5.2 ha95c52a_0 conda-forge ```
/opt/tljh/hub/bin/pip freeze ``` aiohttp==3.8.1 aiosignal==1.2.0 alembic==1.7.7 anyio==3.5.0 argon2-cffi==21.3.0 argon2-cffi-bindings==21.2.0 async-generator==1.10 async-timeout==4.0.2 asynctest==0.13.0 attrs==21.4.0 Babel==2.9.1 backcall==0.2.0 backoff==1.11.1 bcrypt==3.2.0 beautifulsoup4==4.11.1 bleach==5.0.0 certifi==2021.10.8 certipy==0.1.3 cffi==1.15.0 charset-normalizer==2.0.12 cryptography==36.0.2 debugpy==1.6.0 decorator==5.1.1 deepmerge==1.0.1 defusedxml==0.7.1 entrypoints==0.4 escapism==1.0.1 fastjsonschema==2.15.3 frozenlist==1.3.0 greenlet==1.1.2 idna==3.3 importlib-metadata==4.11.3 importlib-resources==5.6.0 ipykernel==6.13.0 ipython==7.32.0 ipython-genutils==0.2.0 jedi==0.18.1 Jinja2==3.1.1 json5==0.9.6 jsonschema==4.4.0 jupyter-client==7.2.2 jupyter-core==4.9.2 jupyter-server==1.16.0 jupyter-telemetry==0.1.0 jupyterhub==1.5.0 jupyterhub-configurator @ git+https://github.com/yuvipanda/jupyterhub-configurator@317759e17c8e48de1b1352b836dac2a230536dba jupyterhub-firstuseauthenticator==1.0.0 jupyterhub-idle-culler==1.2.1 jupyterhub-ldapauthenticator==1.3.2 jupyterhub-nativeauthenticator==1.0.5 jupyterhub-systemdspawner==0.15.0 jupyterhub-tmpauthenticator==0.6 jupyterhub-traefik-proxy==0.3.0 jupyterlab==3.3.3 jupyterlab-pygments==0.2.0 jupyterlab-server==2.12.0 ldap3==2.9.1 Mako==1.2.0 MarkupSafe==2.1.1 matplotlib-inline==0.1.3 mistune==0.8.4 multidict==6.0.2 nbclassic==0.3.7 nbclient==0.5.13 nbconvert==6.5.0 nbformat==5.3.0 nest-asyncio==1.5.5 notebook==6.4.10 notebook-shim==0.1.0 oauthenticator==14.2.0 oauthlib==3.2.0 onetimepass==1.0.1 packaging==21.3 pamela==1.0.0 pandocfilters==1.5.0 parso==0.8.3 passlib==1.7.4 pexpect==4.8.0 pickleshare==0.7.5 pluggy==1.0.0 prometheus-client==0.14.1 prompt-toolkit==3.0.29 psutil==5.9.0 ptyprocess==0.7.0 pyasn1==0.4.8 pycparser==2.21 pycurl==7.45.1 Pygments==2.11.2 pyOpenSSL==22.0.0 pyparsing==3.0.8 pyrsistent==0.18.1 python-dateutil==2.8.2 python-json-logger==2.0.2 pytz==2022.1 pyzmq==22.3.0 requests==2.27.1 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.6 Send2Trash==1.8.0 six==1.16.0 sniffio==1.2.0 soupsieve==2.3.2 SQLAlchemy==1.4.35 terminado==0.13.3 the-littlest-jupyterhub @ git+https://github.com/jupyterhub/the-littlest-jupyterhub.git@0572976d25c18bc88d936f8bd8627bbfed069b53 tinycss2==1.1.1 toml==0.10.2 tornado==6.1 traitlets==5.1.1 typing_extensions==4.1.1 urllib3==1.26.9 wcwidth==0.2.5 webencodings==0.5.1 websocket-client==1.3.2 yarl==1.7.2 zipp==3.8.0 ```
hyamanieu commented 2 years ago

As this is an Issue which normally would require a PR to correct, I'll refine my needs:

Thank you

kgleason commented 2 years ago

I think I am seeing a similar issue. I'll admit upfront that I am new to JuptyerHub, and by extension TLJH.

I was trying to add the bits for AzureAD authentication to the config. I kept getting exceptions on startup about not being able to import jwt. But when I was in the venv, pip freeze | grep -i jwt revealed that both jwt and PyJWT were installed. Running the same pip from outside of the venv only showed PyJWT installed.

This lead me to suspect that oauthenticator, or at least the AzureADAuthenticator class is using the system python rather than the python from within the virtual env.

When did pip3 install jwt as root from outside of the venv, I stopped getting the exception.

But JH still doesn't start and the logging isn't providing anything helpful. I assume that it is more failures from packages due to it using the system python.

I am toying with the idea of snapshotting the VM, and then linking the system python binaries directly into the venv to see if anything changes.

I'm unsure how to proceed, aside from scraping the SSO component.