search

ACCESS-NRI / model-config-tests

Tests for checking model configurations
Apache License 2.0
0 stars 1 forks source link

readme

Model Configuration Pytests and CI

This repository houses pytests that are used as part CI checks for model configurations, as well as reusable components of model configuration CI.

The checksum pytests are used for reproducibility CI checks in this repository. The quick configuration tests are used in any repository that calls config-pr-1-ci.yml or is templated by `ACCESS-NRI/model-configs-template. For example, ACCESS-NRI/access-om2-configs.

Code from these pytests is adapted from COSIMAS's ACCESS-OM2's bit reproducibility tests.

Pytests

How to run pytests manually on NCI

  1. Load payu module - this provides the dependencies needed to run the model

    module use /g/data/vk83/modules
module load payu/1.1.4

  2. Create and activate a python virtual environment for installing and running tests

    python3 -m venv <path/to/test-venv> --system-site-packages
source <path/to/test-venv>/bin/activate

  3. Either pip install a released version of model-config-tests,

    pip install model-config-tests==0.0.1

    Or to install model-config-tests in "editable" mode, first clone the repository, and then run pip install from the repository. This means any changes to the code are reflected in the installed package.

    git clone https://github.com/ACCESS-NRI/model-config-tests/ <path/to/model-config-tests>
pip install -e <path/to/model-config-tests>

  4. Checkout an experiment (in this case it is using an ACCESS-OM2 config)

    git clone https://github.com/ACCESS-NRI/access-om2-configs/ <path/to/experiment>
cd <path/to/experiment>
git checkout <branch/tag>

  5. Run the pytests

    model-config-tests

  6. Once done with testing, deactivate the virtual environment, and if the environment is no longer needed, remove the environment

    deactivate
rm -rf <path/to/test-venv> # Deletes the test environment

Pytest Options

The output directory for pytests defaults to $TMPDIR/test-model-repro and contains the following subdirectories:

This output directory also contains files generated by pytests, including the checksum directory which is used as part of reproducibility CI workflows.

To specify a different folder for pytest outputs, use --output-path command flag, for example:

model-config-tests --output-path /some/other/path/for/output

By default, the control directory, e.g. the model configuration to test, is the current working directory. This can be set similarly to above by using the --control-path command flag.

The path containing the checksum file to check against can also be set using --checksum-path command flag. The default is the testing/checksum/historical-<default-model-runtime>hr-checksums.json file which is stored in the control directory.

To run only CI reproducibility checksum tests, use -m checksum, e.g.

model-config-tests -m checksum

To run quick configuration tests, use the config marker. To additionally run ACCESS-OM2 specific quick configuration tests, use access_om2 marker, e.g.:

model-config-tests -m "config or access_om2"

CI/CD

The .github directory contains many different workflows and actions. This section describes how they are used.

CI/CD For This Repository

CI.yml and CD.yml are used to test, package and upload the model-config-tests package that is used by model-configs-style repositories across the ACCESS-NRI. These are the only workflows that run on this repository. The others are reusable workflows called by model-configs-style repositories, among others.

Reusable CI

The config-*.yml, generate-checksums.yml and test-repro.yml workflows are called by model-configs-style repositories to test model configurations. They are stored in this repository to allow a central place to update generic CI used by all model configuration repositories.

Repos That Use Reusable CI

Currently, these repositories make use of the reusable CI:

Below is information on the use of these workflows.

config-pr-*.yml Pipeline

The config-pr-* Pipeline is a series of workflows that govern the testing, ChatOps and merging procedures of pull requests for model configuration repositories, such as ACCESS-NRI/access-om2-configs.

Essentially, these files work on two types of configuration branch pull requests in the model configuration repository. More information on the terminology used in model configuration repositories can be found in the README.md of the ACCESS-NRI/model-configs-template repository. The types of pull requests are:

config-schedule-*.yml Pipeline

The config-schedule-* Pipeline is a series of workflows used to check the reproducibility of certain config tags against themselves, every month. This is used as a kind of canary to make sure that we continue to get the same results on the same deployment targets.

generate-checksums Reusable Workflow

This workflow is used to easily generate the checksums used in the reproducibility checks, for a specific branch of a model configuration repository, if they don't already exist. This is most often used for the initial commit of a checksum to the release-* configuration branch.

test-repro Reusable Workflow

This workflow is used to test the reproducibility of a given model repository against historical checksums, and can be used as a standalone workflow.

Using it has some requirements outside of just filling in the inputs: One must have a valid GitHub Environment (specified by the environment-name input) in the calling repository, that has the following secrets and vars defined: