search

jupyterhub / the-littlest-jupyterhub

Simple JupyterHub distribution for 1-100 users on a single server
https://tljh.jupyter.org
BSD 3-Clause "New" or "Revised" License
1.03k stars 339 forks source link

`jupyterhub.service` process exits on `ubuntu-2004-lts` on initial installation #846

Closed jrdnbradford closed 1 year ago

jrdnbradford commented 1 year ago

Bug description

TLJH errors out after initial installation on ubuntu-2004-lts focal image on Google Cloud. This is relatively new behavior and seems to be associated with a newly released image (2023-01-25). The ubuntu-1804-lts image seems unaffected.

Expected behaviour

jupyterhub.service should start without erring out.

Actual behaviour

A sqlalchemy.exc.ArgumentError causes the process to exit. See logs below.

How to reproduce

Run the install script on ubuntu-2004-lts image.

curl -L https://tljh.jupyter.org/bootstrap.py | sudo python3 - \
  --admin ${ADMIN}:${PASSWORD} \
  --show-progress-page

tljh-config set user_environment.default_app jupyterlab
tljh-config reload hub

Your personal set up

Full environment Here's a `pip freeze` of the environment which should just be what's on the image; I don't `pip install` anything due to the issue. ``` attrs==19.3.0 Automat==0.8.0 blinker==1.4 certifi==2019.11.28 chardet==3.0.4 Click==7.0 cloud-init==22.4.2 colorama==0.4.3 command-not-found==0.3 configobj==5.0.6 constantly==15.1.0 cryptography==2.8 dbus-python==1.2.16 distro==1.4.0 distro-info===0.23ubuntu1 entrypoints==0.3 httplib2==0.14.0 hyperlink==19.0.0 idna==2.8 importlib-metadata==1.5.0 incremental==16.10.1 Jinja2==2.10.1 jsonpatch==1.22 jsonpointer==2.0 jsonschema==3.2.0 keyring==18.0.1 language-selector==0.1 launchpadlib==1.10.13 lazr.restfulclient==0.14.2 lazr.uri==1.0.3 MarkupSafe==1.1.0 more-itertools==4.2.0 netifaces==0.10.4 oauthlib==3.1.0 pexpect==4.6.0 pyasn1==0.4.2 pyasn1-modules==0.2.1 PyGObject==3.36.0 PyHamcrest==1.9.0 PyJWT==1.7.1 pymacaroons==0.13.0 PyNaCl==1.3.0 pyOpenSSL==19.0.0 pyrsistent==0.15.5 pyserial==3.4 python-apt==2.0.1 python-debian===0.1.36ubuntu1 PyYAML==5.3.1 requests==2.22.0 requests-unixsocket==0.2.0 SecretStorage==2.3.1 service-identity==18.1.0 simplejson==3.16.0 six==1.14.0 sos==4.4 ssh-import-id==5.10 systemd-python==234 Twisted==18.9.0 ubuntu-advantage-tools==27.12 ufw==0.36 unattended-upgrades==0.1 urllib3==1.25.8 wadllib==1.3.3 zipp==1.0.0 zope.interface==4.7.1 ```
Configuration I don't create a config file.
Logs ```bash Waiting for JupyterHub to come up (1/20 tries) Done! Downloading traefik 1.7.33... Progress page server stopped successfully. Running JupyterHub version 1.5.1 Using Authenticator: firstuseauthenticator.firstuseauthenticator.FirstUseAuthenticator Using Spawner: traitlets.traitlets.UserCreatingSpawner-5.8.1 Using Proxy: jupyterhub_traefik_proxy.toml.TraefikTomlProxy-0+unknown Writing cookie_secret to /opt/tljh/state/jupyterhub_cookie_secret [I 2023-01-27 04:13:12.847 alembic.runtime.migration migration:205] Context impl SQLiteImpl. [I 2023-01-27 04:13:12.848 alembic.runtime.migration migration:208] Will assume non-transactional DDL. [I 2023-01-27 04:13:12.865 alembic.runtime.migration migration:619] Running stamp_revision -> 4dc2d5a8c53c [E 2023-01-27 04:13:12.874 JupyterHub app:2991] Traceback (most recent call last): File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/app.py", line 2988, in launch_instance_async await self.initialize(argv) File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/app.py", line 2522, in initialize self.init_db() File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/app.py", line 1726, in init_db self.session_factory = orm.new_session_factory( File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/orm.py", line 880, in new_session_factory check_db_revision(engine) File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/orm.py", line 771, in check_db_revision current_table_names = set(inspect(engine).get_table_names()) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/inspection.py", line 111, in inspect ret = reg(subject) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 304, in _engine_insp return Inspector._construct(Inspector._init_engine, bind) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 237, in _construct init(self, bind) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/engine/reflection.py", line 248, in _init_engine engine.connect().close() File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 3245, in connect return self._connection_cls(self) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 174, in __init__ self.dispatch.engine_connect(self) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/event/attr.py", line 487, in __call__ fn(*args, **kw) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/event/legacy.py", line 100, in wrap_leg return fn(*conv(*args)) File "/opt/tljh/hub/lib/python3.8/site-packages/jupyterhub/orm.py", line 737, in ping_connection connection.scalar(select([1])) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/sql/_selectable_constructors.py", line 493, in select return Select(*entities) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/sql/selectable.py", line 5219, in __init__ self._raw_columns = [ File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/sql/selectable.py", line 5220, in coercions.expect( File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py", line 413, in expect resolved = impl._literal_coercion( File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py", line 652, in _literal_coercion self._raise_for_expected(element, argname) File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py", line 1143, in _raise_for_expected return super()._raise_for_expected( File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py", line 711, in _raise_for_expected super()._raise_for_expected( File "/opt/tljh/hub/lib/python3.8/site-packages/sqlalchemy/sql/coercions.py", line 536, in _raise_for_expected raise exc.ArgumentError(msg, code=code) from err sqlalchemy.exc.ArgumentError: Column expression, FROM clause, or other columns clause element expected, got [1]. Did you mean to say select(1)? jupyterhub.service: Main process exited, code=exited, status=1/FAILURE jupyterhub.service: Failed with result 'exit-code'. Stopped jupyterhub.service ```
welcome[bot] commented 1 year ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

jmkekala commented 1 year ago

I got the same with the latest Ubuntu 22.04.1 LTS (Jammy Jellyfish) directly from the https://releases.ubuntu.com/22.04/.

How I managed to get it running was by downgrading SQLAlchemy from 2.0 to 1.4.46. So probably the latest version (2.0) that was released on 26th breaks this.

misukin commented 1 year ago

I tried a fresh install of Ubuntu 20.04 and 22.04 with latest updates -> always got the same problem. @jmkekala can you please post your downgrade process? Thanks!

lucaschippai2i commented 1 year ago

I got the same with the latest Ubuntu 22.04.1 LTS (Jammy Jellyfish) directly from the https://releases.ubuntu.com/22.04/.

How I managed to get it running was by downgrading SQLAlchemy from 2.0 to 1.4.46. So probably the latest version (2.0) that was released on 26th breaks this.

Good day @jmkekala , I also tried to downgrade SQLAlchemy to 1.4.46 and then reload the hub, but this didn't solve the problem. Could you post the steps you followed? Thanks!

EDIT: I applied the solution described here and it worked.

Commands to run: pip3 install --upgrade 'SQLAlchemy<2.0.0' tljh-config reload hub

misukin commented 1 year ago

@lucaschippai2i thanks for the link, this worked for me:

/opt/tljh/hub/bin/pip3 install --upgrade 'SQLAlchemy<2.0.0 tljh-config reload hub

jrdnbradford commented 1 year ago

Nice, thanks for this @pnasrat.

pnasrat commented 1 year ago

Note to self it looks like the instructions on dev setup https://tljh.jupyter.org/en/latest/contributing/dev-setup.html might be outdated as they no longer setup the path as the docs mention without uncommenting the development install ENV lines in integration-tests/Dockerfile means following the dev-setup instructions works with my PR and no manual intervention to rerun the installer

lucaschippai2i commented 1 year ago

Note to self it looks like the instructions on dev setup https://tljh.jupyter.org/en/latest/contributing/dev-setup.html might be outdated as they no longer setup the path as the docs mention without uncommenting the development install ENV lines in integration-tests/Dockerfile means following the dev-setup instructions works with my PR and no manual intervention to rerun the installer

Could you clarify? Is it still necessary to uncomment these lines (I did) or not?

pnasrat commented 1 year ago

@lucaschippai2i note I'm still getting oriented in the various jupyterhub projects.

The GitHub actions run integrations tests set the TLJH_BOOTSTRAP_PIP_SPEC to a GitHub ref and don't use Dockerfile. By default the bootstrap script will bootstrap itself from the main branch on tljh repo. To follow the instructions on the dev-setup page with the fix not upstreamed and not have to downgrade SQLAlchemy you can uncomment the lines in the Dockerfile, or alternatively pass them in via the -e option to docker run