HSU-HPC / MaMiCo

The macro-micro-coupling tool for coupled molecular-continuum flow simulation
BSD 3-Clause "New" or "Revised" License
13 stars 5 forks source link

Automated Integration Tests #72

Open rubenhorn opened 1 month ago

rubenhorn commented 1 month ago

Why?

The unit tests run as part of the CI pipeline cannot capture all aspects of MaMiCo.
Therefore, it is necessary to also perform integration tests which invoke the compiled binary using a realistic, non-trivial scenario to verify that the configuration generator, simulation, and visualization output is correct.

What?

For every new commit on the master branch or (?) for every commit on a branch with an open pull request to the master branch, the integration test should be performed, executing the following steps:

  1. Clone the repository and check out the corresponding commit
  2. Compile MaMiCo
  3. Generate the MaMiCo configuration (options TBD)
  4. Execute MaMiCo using MPI and the configuration
  5. Visualize the results
  6. Verify the (visualization) results (using a reference)
  7. Report the results of the test

How?

Execution using GitHub Actions is not feasible using the standard runners due to the number of required MPI ranks and the expected non-trivial runtime of the test, so the automated integration test is executed on an on-site system.
Due to security reasons, the automated test should use a pull-semantic (CRON job polling GitHub).
If an error occurs during the execution of the test or the output deviates from the expectation, a notification should be generated to raise awareness about the corresponding commit. This can be done through email or using the GitHub API.
To avoid repeated tests of the same commit and repeated notification about the same issue, a local database of tested commit hashes should be maintained (A file with simple "key=value" pairs should be sufficient).

Chronological checklist of key features

Checklist of stretch goal features

rubenhorn commented 1 month ago

After some discussion @Thinkpiet and I came to the conclusion that we should perhaps simplify the plan for the automated integration tests somewhat to avoid building a full workflow engine with all the complexity that entails.

The core features can now be boiled down to:

  1. Periodically (e.g.: every 6h) fetch the master branch of the repository.
  2. If the commit hash is different from the last one tested (stored in file), trigger new integration tests. (This is done to avoid re-running tests if there is no change for some time, like during holidays.)
  3. Periodically (using a second CRON job) At the end of the FabMaMiCo job, the current test results are compared to the previous ones (and current become previous).
  4. If the results differ from the previous ones, an email with the test results is sent to all developers.