Open rubenhorn opened 1 month ago
After some discussion @Thinkpiet and I came to the conclusion that we should perhaps simplify the plan for the automated integration tests somewhat to avoid building a full workflow engine with all the complexity that entails.
The core features can now be boiled down to:
master
branch of the repository.
Why?
The unit tests run as part of the CI pipeline cannot capture all aspects of MaMiCo.
Therefore, it is necessary to also perform integration tests which invoke the compiled binary using a realistic, non-trivial scenario to verify that the configuration generator, simulation, and visualization output is correct.
What?
For every new commit on the
master
branch or (?) for every commit on a branch with an open pull request to themaster
branch, the integration test should be performed, executing the following steps:How?
Execution using GitHub Actions is not feasible using the standard runners due to the number of required MPI ranks and the expected non-trivial runtime of the test, so the automated integration test is executed on an on-site system.
Due to security reasons, the automated test should use a pull-semantic (CRON job polling GitHub).
If an error occurs during the execution of the test or the output deviates from the expectation, a notification should be generated to raise awareness about the corresponding commit. This can be done through email or using the GitHub API.
To avoid repeated tests of the same commit and repeated notification about the same issue, a local database of tested commit hashes should be maintained (A file with simple "key=value" pairs should be sufficient).
Chronological checklist of key features
Input: Commit hash
Output: Test result (passed/failure details), artifact (visualization)
Details: Cloning the repository and checking out the commit hash, generating the configuration, executing MaMiCo, executing the visualization, invoking a callback with the results (next step of the pipeline)
Input: None
Output: None
Details: Periodically check the state of the remote repository against the local database of executed tests, execute outstanding tests and update the database accordingly (status: started)
Input: Test result, artifact Output: None Details: If the result is not "passed", send a notification including the failure details and artifact (e.g.:
mail
containing details and path to artifact`), and update the database (status: success/failed[-final])Checklist of stretch goal features
git clone
)Rather than sending and email, use the GitHub HTTP API to attach test results as comments to existing pull requests or open a new issue for commits on master.
Replace the simple "key=value" file for storing the test states with an SQLite or DuckDB instance.
Delete files associated with a single test run after a certain amount of time (no sooner than 1 month)