Migration scripts? `no such column: jobs.package_input_folder`

krassowski commented 6 months ago

v2.6.0 release some time ago added a new column package_input_folder https://github.com/jupyter-server/jupyter-scheduler/pull/510 but after upgrade it seems that the extension fails for me with:

Description

│         return self._iter().one()                                                                                                                                                                                                                                                                                                                                                                                                      │
│                ^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                            │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/orm/query.py", line 2916, in _iter                                                                                                                                                                                                                                                                                                                         │
│         result = self.session.execute(                                                                                                                                                                                                                                                                                                                                                                                                 │
│                  ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                 │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1717, in execute                                                                                                                                                                                                                                                                                                                     │
│         result = conn._execute_20(statement, params or {}, execution_options)                                                                                                                                                                                                                                                                                                                                                          │
│                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                          │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20                                                                                                                                                                                                                                                                                                                 │
│         return meth(self, args_10style, kwargs_10style, execution_options)                                                                                                                                                                                                                                                                                                                                                             │
│                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                             │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection                                                                                                                                                                                                                                                                                                      │
│         return connection._execute_clauseelement(                                                                                                                                                                                                                                                                                                                                                                                      │
│                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                      │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement                                                                                                                                                                                                                                                                                                      │
│         ret = self._execute_context(                                                                                                                                                                                                                                                                                                                                                                                                   │
│               ^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                                                                                                                                                   │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context                                                                                                                                                                                                                                                                                                            │
│         self._handle_dbapi_exception(                                                                                                                                                                                                                                                                                                                                                                                                  │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception                                                                                                                                                                                                                                                                                                     │
│         util.raise_(                                                                                                                                                                                                                                                                                                                                                                                                                   │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/util/compat.py", line 211, in raise_                                                                                                                                                                                                                                                                                                                       │
│         raise exception                                                                                                                                                                                                                                                                                                                                                                                                                │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context                                                                                                                                                                                                                                                                                                            │
│         self.dialect.do_execute(                                                                                                                                                                                                                                                                                                                                                                                                       │
│       File "/opt/conda/envs/default/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute                                                                                                                                                                                                                                                                                                                │
│         cursor.execute(statement, parameters)                                                                                                                                                                                                                                                                                                                                                                                          │
│     sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such column: jobs.package_input_folder                                                                                                                                                                                                                                                                                                                              │
│     [SQL: SELECT count(*) AS count_1                                                                                                                                                                                                                                                                                                                                                                                                   │
│     FROM (SELECT jobs.runtime_environment_name AS jobs_runtime_environment_name, jobs.runtime_environment_parameters AS jobs_runtime_environment_parameters, jobs.compute_type AS jobs_compute_type, jobs.input_filename AS jobs_input_filename, jobs.output_formats AS jobs_output_formats, jobs.name AS jobs_name, jobs.tags AS jobs_tags, jobs.parameters AS jobs_parameters, jobs.email_notifications AS jobs_email_notifications, │
│     FROM jobs) AS anon_1]                                                                                                                                                                                                                                                                                                                                                                                                              │
│     (Background on this error at: https://sqlalche.me/e/14/e3q8)                                                                                                                                                                                                                                                                                                                                                                       │

Reproduce

run some jobs in previous versions
upgrade 3, try to open the extension after upgrade

Expected behavior

Either:

there is a migration script
there is an autofixer for database schemas

Context

Jupyter Server version: 4.x

welcome[bot] commented 6 months ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

krassowski commented 6 months ago

For example, jupyterhub uses alembic for migrating database between versions (https://github.com/jupyterhub/jupyterhub/tree/main/jupyterhub/alembic/versions).

CC @andrii-i as author of https://github.com/jupyter-server/jupyter-scheduler/pull/510

dlqqq commented 6 months ago

@andrii-i is working on this right now in #520. Thank you Mike for opening an issue to track this.

andrii-i commented 6 months ago

@krassowski thank for opening this issue. I just released v2.7.0 that among other things adds a database migration / table extension logic (#520).

jupyter-server / jupyter-scheduler