search

E3SM-Project / zppy

E3SM post-processing toolchain
BSD 3-Clause "New" or "Revised" License
6 stars 15 forks source link

Add minimal case cfgs #608

Closed forsyth2 closed 4 months ago

forsyth2 commented 5 months ago

Add minimal case cfgs. Addresses 'Automated testing of the "combinatorial explosion" of parameter options. Often we need to know how zppy behaves when different combination of parameters are applied. Setting up automatic cfg generation and testing of these cases would be immensely helpful.' in #572.

This pull request aims to add some important "minimal cases" -- i.e., cfgs with as few parameters as possible to test specific cases.

forsyth2 commented 4 months ago

I've broken up the complete_run test into components or "minimal cases" as I call them here.

I've also moved the input over from v2 to v3, in some sense resolving #552. To fully resolve #552, we'd need to have an actual test call (as #604 would do with the complete_run test) rather than just producing output (as these cfgs do).

My plan going forward is to use these minimal case cfgs to test future pull requests -- only running the relevant cfgs. E.g., no sense running ILAMB if I'm changing global_time_series code.

As for testing the main branch on some recurring basis, we need to either A) automate running every single one of these minimal cases and somehow checking their results, or B) convert the complete_run tests from v2 to v3 so we have some method of testing main. (A) is more comprehensive, (B) is more practical.

Now, two caveats:

  1. I haven't actually run these minimal cases because there are quite a few of them. My plan is to modify them as needed when actually using them to debug/test future pull requests.
  2. The v3 input is currently hard coded to a simulation on Chrysalis. We'll need to copy over the input data to Perlmutter and Compy (or be content running these minimal cases on Chrysalis only).