CI tests for merges

[kpamnany]

Description

This issue will be closed when we are satisfied that the CI tests for merges (not including daily or weekly tests) are both comprehensive and minimal. To this end, there must be sufficient unit tests for each ClimateMachine module, and a small set of integration tests.

Integration tests

A framework for model testing needs to be built to simplify integration testing and make our integration tests leaner. Please volunteer to lead/participate in this effort.

Unit tests

Below is a list of every module in the ClimateMachine with the checkbox indicating whether there are any unit tests specified for the module together with a note about its current testing state.

Please:

• comment with changes/updates to the contents of this list; I'll update this post periodically.
• claim modules you wrote/understand; you will be "inspector" of those modules, responsible for evaluating the quality of their unit tests.
• open issues for problems with unit tests for the modules you have claimed; these issues will be linked into this one to track progress.

Modules

• [ ] ClimateMachine Driver (top-level: initialization, settings, configuration, run simulation) -- @kpamnany Some indirectly tested by experiments. Need to ensure that all parts are tested.

  730.

  • [ ] Callbacks (periodic functionality, controlled from the Driver: VTK, diagnostics, etc.) -- @kpamnany Some indirectly tested by experiments. Need to ensure that all callbacks are tested.
• [ ] BalanceLaw (abstract type and interface for models: state types, etc.) N/A?
• [ ] Problem (abstract type and interface for IVP) -- @kpamnany N/A?
• [X] MPIStateArray Verify.
  • [ ] CMBuffer
• [ ] Orientations Indirectly tested by experiments.
• [ ] TurbulenceClosures -- @akshaysridhar Currently tested with experiments only. Unit tests are planned.
• [ ] TurbulenceConvection
• [X] SystemSolvers -- @thomasgibson Verify.
• [X] ODESolvers --@thomasgibson Verify.
  • [X] GenericCallbacks
• [X] Mesh Verify.
  • [X] BrickMesh
  • [X] Elements
  • [X] Filters
  • [X] Geometry
  • [X] Grids
  • [X] Interpolation
  • [X] Topologies
• [X] DGMethods Verify.
  • [X] NumericalFluxes
  • [X] Courant
• [ ] Atmos Existing tests appear to be integration tests? Seem incomplete.
  • [X] Microphysics -- @trontrytel Complete. Integration tests use a 2D Kinematic Model -- useful for other integration tests?
  • [X] TemperatureProfiles Verify.
• [ ] Ocean All tests appear to be integration tests?
• [x] Land -- @kmdeck Unit tests could be improved, and possibly some things tested by the integration tests can be made into unit tests (for example, the test_bc and prescribed_twice files).

  1643.

• [ ] Diagnostics -- @kpamnany Indirectly tested in various experiments.
  • [X] StateCheck
• [X] Thermodynamics -- @charleskawczynski Verify.
• [ ] ConfigTypes N/A?
• [ ] Spectra -- @LenkaNovak Indirectly tested in an experiment.
• [X] VariableTemplates Verify.
• [X] SurfaceFluxes/UniversalFunctions -- @charleskawczynski Fix "FIXME". Verify.
• [X] SingleStackUtils (helper functions for single stack configurations) Generalize? GPUify?
• [X] VTK Verify.
• [X] Writers -- @kpamnany Complete.
• [X] Checkpoint -- @kpamnany Unit tests are in the wrong place (test/Driver) and heavyweight.
• [X] TicToc (performance measurement utility) -- @kpamnany Complete.
• [x] ArtifactWrappers Indirectly tested.

[trontrytel]

@kpamnany - Thank you for doing this.

For Microphysics.jl and Microphysics_0M.jl parameterisation modules the unit tests are in runtests.jl I think every parameterisation function has at least one unit test. The tests could be better, though.

The remaining tests use a 2D Kinematic Model setup. This was my attempt to test some of the microphysics separately from model dynamics issues. They are integration tests and are run on buildkite.

[charleskawczynski]

Should we start this by pulling out all files in test/ that rely on invoke!, and move them to (for example) integration_tests/, and see how codecov changes?

[kmdeck]

I checked the Land Box - currently for soil we have two files with true unit tests, and then the rest are integration tests. I think our unit tests could be improved, and possibly some things tested by the integration tests can be made into unit tests (for example, the test_bc and prescribed_twice files)

[simonbyrne]

One thing I would like to see is a test harness for balance laws to check fluxes, sources, boundary conditions etc at single points (i.e. without a DGModel)

[kmdeck]

One thing I would like to see is a test harness for balance laws to check fluxes, sources, boundary conditions etc at single points (i.e. without a DGModel)

this might let us get rid of our integration test test_bc.jl, as the only purpose of that is to test that the values at the boundary throughout the integration match (within some error) the values we supplied as conditions.

[akshaysridhar]

TurbulenceClosures is based on full experiment integration tests currently, but I will refactor those into smaller function-level checks.

[kpamnany]

I checked the Land Box - currently for soil we have two files with true unit tests, and then the rest are integration tests. I think our unit tests could be improved, and possibly some things tested by the integration tests can be made into unit tests (for example, the test_bc and prescribed_twice files)

@kmdeck: would you open an issue in which you describe these improvements to the unit tests? I'll link that issue here for tracking. No commitment is currently necessary on when those improvements will be carried out. :-)

[jiahe23]

TurbulenceClosures is based on full experiment integration tests currently, but I will refactor those into smaller function-level checks.

I have been working on refactoring the hyperdiffusion box test and adding a spherical surface test. Primary modifications in the box test include:

• removing the timestepper and directly comparing RHS from DG model to the analytical solution;
• setting a threshold of 5% for the relative error instead of approximating to some expected values.

There are still some issues with the spherical surface test and @mwarusz is digging into it.

Hope this is helpful, and let me know if there is anything I could help with!

[charleskawczynski]

I'll for sure cover

• Thermodynamics
• SurfaceFluxes / UniversalFunctions (in Common/)

And I'll try to tac on more soon.

[charleskawczynski]

Reviewing CodeCov for the Thermodynamics module, it looks like most of the "missed" coverage is due to inlining. Hoping this will improve with fixes in Julia 1.6