Automated Integration Tests

[rubenhorn]

Why?

The unit tests run as part of the CI pipeline cannot capture all aspects of MaMiCo.
Therefore, it is necessary to also perform integration tests which invoke the compiled binary using a realistic, non-trivial scenario to verify that the configuration generator, simulation, and visualization output is correct.

What?

For every new commit on the master branch or (?) for every commit on a branch with an open pull request to the master branch, the integration test should be performed, executing the following steps:

1. Clone the repository and check out the corresponding commit
2. Compile MaMiCo
3. Generate the MaMiCo configuration (options TBD)
4. Execute MaMiCo using MPI and the configuration
5. Visualize the results
6. Verify the (visualization) results (using a reference)
7. Report the results of the test

How?

Execution using GitHub Actions is not feasible using the standard runners due to the number of required MPI ranks and the expected non-trivial runtime of the test, so the automated integration test is executed on an on-site system.
Due to security reasons, the automated test should use a pull-semantic (CRON job polling GitHub).
If an error occurs during the execution of the test or the output deviates from the expectation, a notification should be generated to raise awareness about the corresponding commit. This can be done through email or using the GitHub API.
To avoid repeated tests of the same commit and repeated notification about the same issue, a local database of tested commit hashes should be maintained (A file with simple "key=value" pairs should be sufficient).

Chronological checklist of key features

• [ ] Integration test execution using FabMaMiCo:
  Input: Commit hash
  Output: Test result (passed/failure details), artifact (visualization)
  Details: Cloning the repository and checking out the commit hash, generating the configuration, executing MaMiCo, executing the visualization, invoking a callback with the results (next step of the pipeline)
• [ ] CRON job:
  Input: None
  Output: None
  Details: Periodically check the state of the remote repository against the local database of executed tests, execute outstanding tests and update the database accordingly (status: started)
• [ ] Reporting the results:
  Input: Test result, artifact Output: None Details: If the result is not "passed", send a notification including the failure details and artifact (e.g.: mail containing details and path to artifact`), and update the database (status: success/failed[-final])

Checklist of stretch goal features

• [ ] (Manually) re-run jobs if they failed due to e.g. network connectivity issues (git clone)
• [ ] Report through GitHub API:
  Rather than sending and email, use the GitHub HTTP API to attach test results as comments to existing pull requests or open a new issue for commits on master.
• [ ] Propper database:
  Replace the simple "key=value" file for storing the test states with an SQLite or DuckDB instance.
• [ ] Housekeeping:
  Delete files associated with a single test run after a certain amount of time (no sooner than 1 month)

[rubenhorn]

After some discussion @Thinkpiet and I came to the conclusion that we should perhaps simplify the plan for the automated integration tests somewhat to avoid building a full workflow engine with all the complexity that entails.

The core features can now be boiled down to:

1. Periodically (e.g.: every 6h) fetch the master branch of the repository.
2. If the commit hash is different from the last one tested (stored in file), trigger new integration tests. (This is done to avoid re-running tests if there is no change for some time, like during holidays.)
3. Periodically (using a second CRON job) At the end of the FabMaMiCo job, the current test results are compared to the previous ones (and current become previous).
4. If the results differ from the previous ones, an email with the test results is sent to all developers.