The nix tests in this repo have gotten fairly time consuming. For example a recent workflow took an hour and five minutes. This is more than double the typical amount of time required for the docker tests (see a comparable workflow here). This PR breaks the nix tests up into a few categories:
basic: these are tests that will be run upon every push to every PR. These represent something of a bare minimum requirement: the model and wrapper build successfully, answers don't change for our default regression test, and the model restarts in a reproducible fashion.
coarse: these are tests that pertain to coarse-graining code.
emulation: these are tests that pertain to emulation code.
unmarked: these are miscellaneous tests that relate to the fortran code, but are not likely to be impacted by every PR.
wrapper: these are tests of the Python wrapper.
Only the "basic" tests are run automatically in each PR. The other categories of tests can be launched by releasing their respective CircleCI holds; through caching they will automatically take advantage of the code compiled for the basic tests to save time and resources. When a PR is merged to master, all tests will be run automatically just to make completely sure nothing has broken. This is a similar strategy to what we use in fv3net regarding the integration tests.
After this refactor, the docker tests are the rate limiting step in CI, taking about 30 minutes. The basic nix tests take about 20 minutes to complete.
The nix tests in this repo have gotten fairly time consuming. For example a recent workflow took an hour and five minutes. This is more than double the typical amount of time required for the docker tests (see a comparable workflow here). This PR breaks the nix tests up into a few categories:
Only the "basic" tests are run automatically in each PR. The other categories of tests can be launched by releasing their respective CircleCI holds; through caching they will automatically take advantage of the code compiled for the basic tests to save time and resources. When a PR is merged to master, all tests will be run automatically just to make completely sure nothing has broken. This is a similar strategy to what we use in fv3net regarding the integration tests.
After this refactor, the docker tests are the rate limiting step in CI, taking about 30 minutes. The basic nix tests take about 20 minutes to complete.