sematic-ai / sematic

An open-source ML pipeline development platform
Other
975 stars 59 forks source link

Switch to SQLAlchemy 2.0 #1127

Closed augray closed 3 months ago

augray commented 3 months ago

SQLAlchemy 2.0 came out in January of 2023 (>1.5 years ago). It introduced a variety of breaking changes. Sematic uses SQLAlchemy heavily, and in complex ways (ORM type introspection for dynamic serialization/deserialization, ORM type mixins, complex and dynamic query builds). This meant we had to wait to upgrade until sufficient time could be dedicated to migrating and testing it. We don't want to wait too much longer, it's only a matter of time before 1.x releases stop getting published and SQLAlchemy's development has largely shifted to >=2.0. So this PR bites the bullet and does the upgrade.

This PR also upgrades the server image python version to 3.12. While not technically related to the SQLAlchemy changes, it follows closely on the heels of #1126 , and since that did not change anything running on the server (but this does) it seemed like an OK place to add so the 3.12 server and SQLAlchemy 2.0 changes could be tested together.

Finally, there was actually a bug for custom metrics usage that was not caught in unit tests. This PR fixes that bug, confirmed in remote testing.

Testing

In addition to all unit tests passing, I also tested running a local server with bazel and running the MNIST example in it. I confirmed built-in metrics worked and the pipeline passed with the expected resulting UI.

I then built a wheel with these changes and published a release candidate Docker image for the standard and EE images.

In a remote environment, I: