ai2cm / fv3gfs-fortran

FV3GFS Fortran for internal development at AI2
Other
6 stars 9 forks source link

Defer slow (not necessarily essential) tests to after merge #339

Closed spencerkclark closed 2 years ago

spencerkclark commented 2 years ago

The nix tests in this repo have gotten fairly time consuming. For example a recent workflow took an hour and five minutes. This is more than double the typical amount of time required for the docker tests (see a comparable workflow here). This PR breaks the nix tests up into a few categories:

Only the "basic" tests are run automatically in each PR. The other categories of tests can be launched by releasing their respective CircleCI holds; through caching they will automatically take advantage of the code compiled for the basic tests to save time and resources. When a PR is merged to master, all tests will be run automatically just to make completely sure nothing has broken. This is a similar strategy to what we use in fv3net regarding the integration tests.

After this refactor, the docker tests are the rate limiting step in CI, taking about 30 minutes. The basic nix tests take about 20 minutes to complete.

spencerkclark commented 2 years ago

Good thought regarding making the basic nix tests a required check. I don't have owner access, but I'll inquire on Slack about making that happen.