Closed jo-basevi closed 5 months ago
This is a really neat approach @jo-basevi
I'd definitely like to get some more eyes on this and get some feedback.
If you have time @dougiesquire it would be good to get a model dev perspective, even if it's "not really my problem, I trust you're doing the right thing".
So more specific: I agree it makes the testing configuration more explicit, easier to understand and in a single location.
@dougiesquire we'll be moving a lot of the infra into the model-configs-agnostic model-configs-test
repo once merged: see https://github.com/ACCESS-NRI/model-config-tests/issues/23
Pytests for QA and reproducibility are going to be packaged up in
model-config-tests
.Related PR to repro workflow (https://github.com/ACCESS-NRI/reproducibility/pull/28) - adds
model-config-tests-version
andpython-version
as inputs.I initially added a bunch of
model-config-tests-version
andpython-version
in the CI workflow files. This makes updating versions error-some. As it would be good just to change the versions in one place, we could have aconfig/ci.json
file on themain
branch that stores all this information. Then thinking about what this could like, it also makes sense to store any information needed to runmodel-config-tests
, e.g. test markers. Custom test markers can be set in the branch workflow code, but this is difficult to find as can't easily search the repository for it. Another positive is that it makes the workflows (slightly) more generalised, so it'll be easier to make them reusable.The config file could like:
I've added defaults as most checks will use the same
model-config-tests-version
,python-version
fields. The keys inscheduled
are tags to run scheduled checks on, while keys in thereproducibility
andqa
would be target branches. Ifmodel-config-tests-version
/markers
/python-version
is not specified for a branch/tag, fallback todefault
for the test type, and then the globaldefault
.Closes #109