[FEATURE REQUEST] Add complete set of integration tests

[yantosca]

The GEOS-Chem integration tests have in general been working well and have allowed us to catch and fix several issues in both GCClassic and GCHP before releasing a new version.

However, some recent "edge cases" (cf https://github.com/geoschem/HEMCO/issues/82, https://github.com/geoschem/HEMCO/issues/84) were not caught by the integration tests and thus required subsequent bug fix commits.

The following additional integration tests would be very useful:

• [x] Nested grid integration tests (at least one)
• [ ] Running across the end of a year (e.g. start at 00z Dec 31 and run to 01z Jan 1st)
• [ ] GCHP stretched grid integration tests
• [x] GCC parallelization tests (single-core vs multi-core)
• [x] Difference tests for zero-diff updates (should be optional)
• [x] Integration test with BPCH_DIAG on (or better yet remove BPCH_DIAG!)
• [x] Integration test for GCHP RRTMG
• Other structural updates
  • [x] Make sure that GCHP end-of-run restart files are renamed and saved to the Restarts/ rundir folder for each test (see #1794)
  • [x] Make sure that GCClassic HEMCO_diagnostics*nc are saved to the OutputDir/ rundir folder for each test (see #1794)
• [x] Add option to have integration tests "skip bootstrapping" of missing restart file variables. This can be useful to do right before a release to see if we need to post new restart files from 1-yr benchmarks, especially when new species have been added.

@lizziel, @msulprizio, @yidant: feel free to add to this list

[LiamBindle]

I've ran into an issue a couple times where ExtData.rc was missing an update in HEMCO_Config.rc. If we're putting together a wishlist, I would add something to check that ExtData.rc and HEMCO_Config.rc are in agreement.

Note: I haven't ran into this with official releases, but often I work with developement code and I've ran into it there.

[lizziel]

Looks good. I think once we have zero diffs across single and multi runs we could have a regular test for that too. We aren't there yet though.

@LiamBindle, regarding your request, we've put that on hold pending the update of ExtData.rc and HEMCO_Config.rc to yaml. However, if we find an eager student to put together a script to check the config files as they are now then we should definitely encourage that.

[lizziel]

I just expanded the list to include GCHP RRTMG.

[yantosca]

I've re-enabled the nested-grid integration tests. This was made possible by creating new sample boundary condition files as described in #811.

[yantosca]

Also note: integration tests now use default run directory names in PR #1565

[yantosca]

I've added a GCHP RRTMG integration test as part of #1565.

[yantosca]

@lizziel @msulprizio: Do we still need "DIfference tests for zero-diff updates?" We now have that implmented via the 1-hr cloud benchmarks.

[lizziel]

I added a dry run integration test to the list.