This is the first of ten PRs for the rewrite of alpenhorn.
Also, just to be clear, the only reason these PRs are coming from a personal fork into the main repo are so I can spam-rewrite the history over and over in GitHub without flooding the #pipeline channel on Slack.
Overview
This PR starts out by doing some maintenance, and then does updates db.py and extensions.py to integrate with the new chimedb.core.alpenhorn module I've introduced over in https://github.com/chime-experiment/chimedb/pull/34.
Housekeeping
The housekeeping work done to prime this re-write is as follows:
minimum Python version has been raised from 3.6 to 3.9. The CI workflow is now run on version 3.9 and 3.11 instead of 3.6 and 3.9.
I have deleted the /legacy/ directory which contained some of the code from the original alpenhorn-1 implementation
I have deleted all of the YAML-based test data which was in the tests/fixtures directory. This has broken the marjority of existsing tests, which I've simply disabled for now. As more parts of the code gets re-written, I'll be introducing fixes to the test suite to cover the new code.
I've also moved all the client tests to a tests/client subdirectory to mirror the organisation of the code itself.
Running black has caused a reflow of text all over the place.
Changes to what a database extension provides
The essential change here is that instead the register_extension function of a database extension returning an function
that initialises peewee.Database, it instead returns a dict of "capabilities", which includes a key called "connect" key that provides the peewee.Database-making function.
The capability dict also contains a close function to close the database (probably unnecessary) and a boolean indicating whether the database extension is threadsafe or not.
If no database extension module is provided, or one or more of these keys is missing from the capability dict, then alpenhorn's db.py provides fallback implementations of all of these. The fallback is not threadsafe.
(Implementation note: I originally had everything that chimedb.core could possibly provide in this capability dict, but as the alpenhorn code matured, I came to the conclusion that more and more of it was more trouble than it was worth, and I've now pared it down to this minimal set.)
The function extensions.connect_database_extension has largely been replaced by db.init which sets-up the db module based on whatever database extension was found (if any). The extensions module also now complains if more than one database extension is specified in the config (since only one of them could possibly be used).
Operationally, the changes here mean that alpenhorn start-up now needs to have a
db.init()
call after extensions are loaded but before the first db.connect() call to set up the db module. Additionally, each thread that needs to access the database also needs to call db.connect() before doing that.
Peewee-3 changes
I have ported the fixes made in chimedb.core to both the mixin RetryOperationalError and the EnumField class so that they work in peewee-3. See https://github.com/chime-experiment/chimedb/pull/34 for details. In practice, when using the CHIME database extension, we don't use the alpenhorn-provided RetryOperationalError. We do use the EnumField defined here, but don't need the fix because MySQL has a native Enum type making the fix unnecessary.
#pipeline
channel on Slack.Overview
This PR starts out by doing some maintenance, and then does updates
db.py
andextensions.py
to integrate with the newchimedb.core.alpenhorn
module I've introduced over in https://github.com/chime-experiment/chimedb/pull/34.Housekeeping
The housekeeping work done to prime this re-write is as follows:
/legacy/
directory which contained some of the code from the originalalpenhorn-1
implementationtests/fixtures
directory. This has broken the marjority of existsing tests, which I've simply disabled for now. As more parts of the code gets re-written, I'll be introducing fixes to the test suite to cover the new code.tests/client
subdirectory to mirror the organisation of the code itself.black
has caused a reflow of text all over the place.Changes to what a database extension provides
The essential change here is that instead the
register_extension
function of a database extension returning an function that initialisespeewee.Database
, it instead returns a dict of "capabilities", which includes a key called "connect" key that provides thepeewee.Database
-making function.The capability dict also contains a
close
function to close the database (probably unnecessary) and a boolean indicating whether the database extension is threadsafe or not.If no database extension module is provided, or one or more of these keys is missing from the capability dict, then alpenhorn's
db.py
provides fallback implementations of all of these. The fallback is not threadsafe.(Implementation note: I originally had everything that
chimedb.core
could possibly provide in this capability dict, but as the alpenhorn code matured, I came to the conclusion that more and more of it was more trouble than it was worth, and I've now pared it down to this minimal set.)The function
extensions.connect_database_extension
has largely been replaced bydb.init
which sets-up thedb
module based on whatever database extension was found (if any). Theextensions
module also now complains if more than one database extension is specified in the config (since only one of them could possibly be used).Operationally, the changes here mean that alpenhorn start-up now needs to have a
call after extensions are loaded but before the first
db.connect()
call to set up thedb
module. Additionally, each thread that needs to access the database also needs to calldb.connect()
before doing that.Peewee-3 changes
I have ported the fixes made in
chimedb.core
to both the mixinRetryOperationalError
and theEnumField
class so that they work in peewee-3. See https://github.com/chime-experiment/chimedb/pull/34 for details. In practice, when using the CHIME database extension, we don't use the alpenhorn-providedRetryOperationalError
. We do use theEnumField
defined here, but don't need the fix because MySQL has a native Enum type making the fix unnecessary.