ACCESS-NRI / access-om2-configs

Standard ACCESS-OM2 configurations released and supported by ACCESS-NRI
3 stars 4 forks source link

Update to use model-config-tests and repro test call #111

Closed jo-basevi closed 5 months ago

jo-basevi commented 5 months ago

Pytests for QA and reproducibility are going to be packaged up in model-config-tests.

Related PR to repro workflow (https://github.com/ACCESS-NRI/reproducibility/pull/28) - adds model-config-tests-version and python-version as inputs.

I initially added a bunch of model-config-tests-version and python-version in the CI workflow files. This makes updating versions error-some. As it would be good just to change the versions in one place, we could have a config/ci.json file on the main branch that stores all this information. Then thinking about what this could like, it also makes sense to store any information needed to run model-config-tests, e.g. test markers. Custom test markers can be set in the branch workflow code, but this is difficult to find as can't easily search the repository for it. Another positive is that it makes the workflows (slightly) more generalised, so it'll be easier to make them reusable.

The config file could like:

{
    "$schema": "./ci.schema.json",
    "scheduled": [
        "release-1deg_jra55_ryf-2.0": {},
        "default": 
            {
                "markers": "checksum"
            }
    ],
    "reproducibility": [
        "default": 
            {
                "markers": "checksum"
            }
    ],
    "qa": [
        "default": 
            {
                "markers": "access-om2 or config"
            }
    ],
    "default": {
        "model-config-tests-version": "0.0.1",
        "python-version": "3.11.0"
    }
}

I've added defaults as most checks will use the same model-config-tests-version, python-version fields. The keys in scheduled are tags to run scheduled checks on, while keys in the reproducibility and qa would be target branches. If model-config-tests-version/markers/python-version is not specified for a branch/tag, fallback to default for the test type, and then the global default.

Closes #109

aidanheerdegen commented 5 months ago

This is a really neat approach @jo-basevi

I'd definitely like to get some more eyes on this and get some feedback.

If you have time @dougiesquire it would be good to get a model dev perspective, even if it's "not really my problem, I trust you're doing the right thing".

aidanheerdegen commented 5 months ago

So more specific: I agree it makes the testing configuration more explicit, easier to understand and in a single location.

CodeGat commented 5 months ago

@dougiesquire we'll be moving a lot of the infra into the model-configs-agnostic model-configs-test repo once merged: see https://github.com/ACCESS-NRI/model-config-tests/issues/23