Upgrade to SQLAlchemy 2.x

[mattholy]

Hi all,

Pandas is now depending on SQLAlchemy 2.x When using pandas and jupyterlab at the same time, which is a very common situation, it may case some conflicts.

https://github.com/jupyter-server/jupyter-scheduler/blob/1b50f81f966c9e0146f91b1606fd78519857527e/pyproject.toml#L36

[JasonWeill]

The sqlalchemy 2.x dependency was introduced in pandas 2.2.0, which was released on 2024-01-19.

[andrii-i]

@mattholy thank you for opening this issue. We plan to update to SQLAlchemy 2.x at some point but this is not a priority / we are not working on it now.

[adampinky85]

Hi @andrii-i, thanks for your work on this project, we use it extensively and it's really great!

Is there any chance you are able to upgrade to SQLAlchemy 2.x? We are finding this is a major blocker recently.

Many thanks!

[andrii-i]

Hi @adampinky85. Glad you are finding Jupyter Scheduler useful.

We are still planning to update to SQLAlchemy 2.x at some point but this is not a priority / we are not working on it now. At the same time, could you please share more about why Jupyter Scheduler not using SQLAlchemy 2.x is a major blocker for you recently?

[adampinky85]

thanks @andrii-i, yes we manage environments running jupyterlabs and the kernel is built with a large number dependencies for research purposes including jupyter-scheduler.

On the EC2 instance we also have a number of project that are cloned and available for users to access from their notebooks (e.g., analysis libraries or business logic). There are now a number of libraries that require SQLAlchemy 2.x. which are not compatible to run from our notebooks.

jupyter-scheduler is exactly the right tool for our use case, we don't need airflow / apscheduler etc. as it requires more effort for users. They want to be able to run their notebooks easily with a few clicks.

[andrii-i]

@mattholy, @adampinky85, thank you for bringing up this issue and providing context. I just released v2.7.0 that among other things updates SQLAlchemy to 2.x.

[FacePrintPay]

some of the specific changes made in the SQLAlchemy version update from 1.0 to 2.×:

1. True division and floor division: The Core expression language now supports both "true division" (i.e. the / Python operator) and "floor division" (i.e. the // Python operator) including backend-specific behaviors to normalize different databases in this regard.
2. Improvements to declarative mixins, @declared_attr and related features: The declarative system in conjunction with declared_attr has been overhauled to support new capabilities. A function decorated with declared_attr is now called only after any mixin-based column copies are generated, allowing it to call upon mixin-established columns and receive a reference to the correct Column object.
3. Association proxies: The AssociationProxy class now supports reuse among subclasses, allowing a particular hybrid to be repurposed across subclasses or other classes.
4. Type annotations: SQLAlchemy now includes type annotations for all public APIs, making it easier to use with static type checkers.
5. Performance improvements: SQLAlchemy 2.× includes several performance improvements, including faster query execution and reduced memory usage.
6. Deprecated features: Several features have been deprecated in SQLAlchemy 2.×, including the text() function for constructing SQL expressions, the type_ attribute of Column objects, and the bindparam() function for constructing parameterized queries.

Here is an example of how you might use some of these new features in your code:

python Edit Run Full Screen Copy code

from sqlalchemy import Column, Integer, String, ForeignKey from sqlalchemy.orm import relationship from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.ext.associationproxy import association_proxy

Base = declarative_base()

class A(Base): tablename = "a" id = Column(Integer, primary_key=True) type = Column(String)

__mapper_args__ = {"polymorphic_on": type, "polymorphic_identity": "a"}

class ASub1(A): __mapper_args__ = {"polymorphic_identity": "asub1"}

class ASub2(A): __mapper_args__ = {"polymorphic_identity": "asub2"}

class B(Base): tablename = "b"

id = Column(Integer, primary_key=True) a_id = Column(Integer, ForeignKey(" a.id"))

a = relationship("A", primaryjoin="B.a_id == A.id", backref="b")

b_value = association_proxy("a", "type")

class C(Base): tablename = "c" id = Column(Integer, primary_key=True) b_id = Column(Integer, ForeignKey("b.id"))

b = relationship("B", backref="c")

In this example, we've used the new type annotations for the Column and relationship objects, as well as the new association_proxy class to create a proxy for the type attribute of the A class. We've also used the new polymorphic_identity attribute to define subclasses of the A class.

On Wed, May 29, 2024, 1:13 PM Cygel White @.***> wrote:

[build-system] requires = ["hatchling>=1.3.1", "jupyterlab~=4.0"] build-backend = "hatchling.build"

[project] name = "jupyter_scheduler" version = "2.5.0" description = "A JupyterLab extension for running notebook jobs" readme = "README.md" license = { file = "LICENSE" } requires-python = ">=3.7" authors = [ { name = "Project Jupyter" }, ] keywords = ["Jupyter", "JupyterLab", "JupyterLab3"] classifiers = [ "Framework :: Jupyter", "Framework :: Jupyter :: JupyterLab", "Framework :: Jupyter :: JupyterLab :: 3", "Framework :: Jupyter :: JupyterLab :: Extensions", "Framework :: Jupyter :: JupyterLab :: Extensions :: Prebuilt", "License :: OSI Approved :: BSD License", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", ] dependencies = [ "jupyter_server>=1.6,<3", "traitlets~=5.0", "nbconvert~=7.0", "pydantic>=1.10,<3", "SQLAlchemy>=2.0,<3", "croniter~=1.4", "pytz==2023.3", "fsspec==2023.6.0", "psutil~=5.9" ]

[project.optional-dependencies] test = [ "pytest", "pytest-cov", "jupyter_server[test]>=1.6,<3" ] dev = [ "click" ] docs = [ "sphinx", "myst_parser", "pydata_sphinx_theme" ]

[project.urls] Homepage = "https://github.com/jupyter-server/jupyter-scheduler"

[tool.check-wheel-contents] ignore = ["W002"]

[tool.hatch.build] artifacts = ["jupyter_scheduler/labextension"]

[tool.hatch.build.targets.wheel.shared-data] "jupyter_scheduler/labextension" = @.***/scheduler" "install.json" = "share/jupyter/labextensions/jupyter_scheduler/install.json" "jupyter-config/server-config" = "etc/jupyter/jupyter_server_config.d" "jupyter-config/nb-config" = "etc/jupyter/jupyter_notebook_config.d"

[tool.hatch.build.targets.sdist] exclude = [".github"]

[tool.hatch.build.hooks.jupyter-builder] dependencies = ["hatch-jupyter-builder>=0.5"] build-function = "hatch_jupyter_builder.npm_builder" ensured-targets = [ "jupyter_scheduler/labextension/static/style.js", "jupyter_scheduler//labextension/package.json", ] skip-if-exists = ["jupyter_scheduler/labextension/static/style.js"]

[tool.hatch.build.hooks.jupyter-builder.build-kwargs] build_cmd

On Wed, May 8, 2024, 11:42 PM adampinky85 @.***> wrote:

Hi @andrii-i https://github.com/andrii-i, thanks for your work on this project, we use it extensively and it's really great!

Is there any chance you are able to upgrade to SQLAlchemy 2.x? We are finding this is a major blocker recently.

Many thanks!

— Reply to this email directly, view it on GitHub https://github.com/jupyter-server/jupyter-scheduler/issues/478#issuecomment-2101875995, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZFSY64G6UXUWVPVIWAJD4TZBLWA5AVCNFSM6AAAAABCUD23GKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBRHA3TKOJZGU . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[adampinky85]

@andrii-i thank you, we really appreciate the effort! I've just rolled the upgrade and everything is working great :)

[andrii-i]

Hi @FacePrintPay. I removed your comment that cited content of pyproject.toml as it looked like spam. Please feel free to repost with more context.