jngrad
commented
2 weeks ago The CTest tool is described in more detail in the the "Mastering CMake" book, chapter Testing With CMake and CTest. The CMake community provides detailed instructions on Using CTest and CDash without CMake. I skipped the CDash part, since I don't have much experience with the dashboard and wasn't sure how it would integrate in GitHub Actions. Most pyMBE and ESPResSo users are already familiar with CTest, and its ASCII-based reporting should be sufficient for our needs.
Long tests are sorted by decreasing runtime. This way, the first time make functional_tests is called, the CTest scheduler runs the slowest tests first. This cuts the samples job runtime from 50 min down to 20 min (logfile 29390808749). This only works the first time, though! For end users who run make functional_tests or make tests multiple times, the test order may not match with the order specified in testsuite/CTestTestfile.cmake after the first run. That's because the CTest scheduler keeps track of test statistics in testsuite/Testing/Temporary, such as runtimes and test failures, and uses this information to improve future runs. For example, failed tests in the last run will be scheduled to start first the next run, so as to give users rapid feedback on whether their latest code changes have fixed the failing tests.
While this PR adds CMake as a dependency, I have no intention of making pyMBE a CMake project. CMake is required by CTest, but anyone can edit the testsuite/CTestTestfile.cmake without the need to be familiar with the CMake syntax. In fact, variables like CMAKE_CURRENT_SOURCE_DIR are not even available, since we don't have a CMake configuration stage. This can come as a surprise to experienced CMake users.
Replace the Makefile-based test driver by the CTest driver. Tests can now run in parallel with
make unit_tests -j8.