SQLAlchemy 2.0 came out in January of 2023 (>1.5 years ago). It introduced a variety of breaking changes. Sematic uses SQLAlchemy heavily, and in complex ways (ORM type introspection for dynamic serialization/deserialization, ORM type mixins, complex and dynamic query builds). This meant we had to wait to upgrade until sufficient time could be dedicated to migrating and testing it. We don't want to wait too much longer, it's only a matter of time before 1.x releases stop getting published and SQLAlchemy's development has largely shifted to >=2.0. So this PR bites the bullet and does the upgrade.
This PR also upgrades the server image python version to 3.12. While not technically related to the SQLAlchemy changes, it follows closely on the heels of #1126 , and since that did not change anything running on the server (but this does) it seemed like an OK place to add so the 3.12 server and SQLAlchemy 2.0 changes could be tested together.
Finally, there was actually a bug for custom metrics usage that was not caught in unit tests. This PR fixes that bug, confirmed in remote testing.
Testing
In addition to all unit tests passing, I also tested running a local server with bazel and running the MNIST example in it. I confirmed built-in metrics worked and the pipeline passed with the expected resulting UI.
I then built a wheel with these changes and published a release candidate Docker image for the standard and EE images.
In a remote environment, I:
Confirmed the built-in system pipeline metrics (ex: success rate and timings) worked
ran a pipeline end-to-end
ran a pipeline that used a Ray cluster
ran a pipeline that used custom metrics and confirmed the UI works
SQLAlchemy 2.0 came out in January of 2023 (>1.5 years ago). It introduced a variety of breaking changes. Sematic uses SQLAlchemy heavily, and in complex ways (ORM type introspection for dynamic serialization/deserialization, ORM type mixins, complex and dynamic query builds). This meant we had to wait to upgrade until sufficient time could be dedicated to migrating and testing it. We don't want to wait too much longer, it's only a matter of time before 1.x releases stop getting published and SQLAlchemy's development has largely shifted to >=2.0. So this PR bites the bullet and does the upgrade.
This PR also upgrades the server image python version to 3.12. While not technically related to the SQLAlchemy changes, it follows closely on the heels of #1126 , and since that did not change anything running on the server (but this does) it seemed like an OK place to add so the 3.12 server and SQLAlchemy 2.0 changes could be tested together.
Finally, there was actually a bug for custom metrics usage that was not caught in unit tests. This PR fixes that bug, confirmed in remote testing.
Testing
In addition to all unit tests passing, I also tested running a local server with bazel and running the MNIST example in it. I confirmed built-in metrics worked and the pipeline passed with the expected resulting UI.
I then built a wheel with these changes and published a release candidate Docker image for the standard and EE images.
In a remote environment, I: