Rewrite 1/10: preliminaries and CHIMEDB integration

ketiltrout commented 1 year ago

This is the first of ten PRs for the rewrite of alpenhorn.

Overview

This PR starts out by doing some maintenance, and then does updates db.py and extensions.py to integrate with the new chimedb.core.alpenhorn module I've introduced over in https://github.com/chime-experiment/chimedb/pull/34.

Housekeeping

The housekeeping work done to prime this re-write is as follows:

minimum Python version has been raised from 3.6 to 3.9. The CI workflow is now run on version ~3.9~ 3.10 and 3.11 instead of 3.6 and ~3.9~ 3.10.
I have deleted the /legacy/ directory which contained some of the code from the original alpenhorn-1 implementation
I have deleted all of the YAML-based test data which was in the tests/fixtures directory. This has broken the marjority of existsing tests, which I've simply disabled for now. As more parts of the code gets re-written, I'll be introducing fixes to the test suite to cover the new code.
I've also moved all the client tests to a tests/client subdirectory to mirror the organisation of the code itself.
Running black has caused a reflow of text all over the place.

Changes to what a database extension provides

The essential change here is that instead the register_extension function of a database extension returning an function that initialises peewee.Database, it instead returns a dict of "capabilities", which includes a key called "connect" key that provides the peewee.Database-making function.

The capability dict also contains a close function to close the database (probably unnecessary) and a boolean indicating whether the database extension is threadsafe or not.

If no database extension module is provided, or one or more of these keys is missing from the capability dict, then alpenhorn's db.py provides fallback implementations of all of these. The fallback is not threadsafe.

(Implementation note: I originally had everything that chimedb.core could possibly provide in this capability dict, but as the alpenhorn code matured, I came to the conclusion that more and more of it was more trouble than it was worth, and I've now pared it down to this minimal set.)

The function extensions.connect_database_extension has largely been replaced by db.init which sets-up the db module based on whatever database extension was found (if any). The extensions module also now complains if more than one database extension is specified in the config (since only one of them could possibly be used).

Operationally, the changes here mean that alpenhorn start-up now needs to have a

db.init()

call after extensions are loaded but before the first db.connect() call to set up the db module. Additionally, each thread that needs to access the database also needs to call db.connect() before doing that.

Peewee-3 changes

I have ported the fixes made in chimedb.core to both the mixin RetryOperationalError and the EnumField class so that they work in peewee-3. See https://github.com/chime-experiment/chimedb/pull/34 for details. In practice, when using the CHIME database extension, we don't use the alpenhorn-provided RetryOperationalError. We do use the EnumField defined here, but don't need the fix because MySQL has a native Enum type making the fix unnecessary.

ketiltrout commented 1 year ago

I realised I can't make stacked PRs if they're off in some other fork, so I've remade #143 locally.

ketiltrout commented 1 year ago

Mostly doc updates and type hinting. I've updated the minimum version to 3.10 for type hinting.

I also found some more unittests in tests/test_extensions.py that were supposed to be in this PR but got overlooked the first time round.

Re-blackened for the 2023 version of black.

radiocosmology / alpenhorn