Simple test harness

[robnagler]

Zgoubi has a nice feature that it can read a .res file to get the .dat used to generate it. We can use his feature to build as simple test harness called full_test.sh:

#!/bin/bash
set -euo pipefail
prog=$(cd ../build && pwd)/zgoubi
rm -rf full_work
cd full_work
for f in ../full_data/*.res; do
        echo "$(basename $f .res)"
        cp "$f" zgoubi.dat
        "$prog"
        diff "$f" zgoubi.res
done
echo PASS

We can then add any *.res files in full_work, and they'll be copied to full_data one by one until one fails, and the test will exit without a PASS.

Per @dtabell's suggestion we should probably use ndiff or the newer numdiff.

[zbeekman]

@robnagler It's up to you, but the proposed approach would either make the process appear as 1 test to CTest, or be completely outside of CTest. This is fine, but CTest has some nice bells and whistles that you won't be getting if you use a shell script test harness like this.

Currently, we could easily implement an analogous solution with CTest. We can populate a directory in the build tree or source tree called full_data with .res files, then copy them to full_work/zgoubi.dat.

Current impediments to this approach are:

• Needing to modify PARIZ.H and recompile the binary depending on the test input
• Requiring different additional assets for the tests, like warmsnake.map
• It appears you cannot diff the entire file, because there is some additional output at the end of the file that may be unique to the run. ndiff appears to be easier to install and more widely available, FWIW. We may still need to solve some issues in parsing out numeric values from text, but I have no experience with ndiff or numdiff, so I may be mistaken here.

Modernizing zgoubi to use dynamic memory allocation will fix the first issue. I'm not sure how many tests require additional files beyond zgoubi.dat, so I can't speak to the difficulty of solving the second point. @dtabell may be able to weigh in here.

[zbeekman]

Reading @robnagler's comment here I think it might make more sense to create one directory per test in the full_work directory, so that we can see all the inputs and outputs, rather than running all the tests in the same directory. (Also easily accomplished with CMake.)

[robnagler]

What are the bells and whistles of CTest?

[zbeekman]

CTest has quite a long list of features, however the most immediately beneficial and useful (also, CTest was explicitly part of the SOW1) are:

• Parallel execution of tests, with the ability to allocate a number of processor per-test; important when testing parallel programs. (Multiple tests can execute simultaneously ensuring enough processors are allocated to each)
• Random scheduling. Useful sometimes for catching uninitialized variables since uninitialized memory may take different values from run to run; if always run in the same order then frequently the previous test will consistently write to the same memory locations giving a seemingly predictable initial value to un-initialized memory
• Repeat N times or until failure option is useful (often combined with the random scheduling) to locate race conditions in parallel code that infrequently manifest themselves
• Total time of tests (split by label)
• Selection of tests to run via regex or number
• Integration with a CDash dashboard if you are supporting multiple clients/clusters/configurations to give overviews of nightly or CI testing
• Tight integration with CMake allows for
  • tweaking tests based on build configuration; this is particularly useful for MPI, since the launcher may take different names and require different flags depending on the hardware and MPI distribution. (CMake's FindMPI module abstracts away this giant headache.)
  • Configuring, building, and testing with a single command
  • Managing test assets and data (in a portable way, to boot) with CMake's downloading, hashing, and more abstract data management like the DATA{...} statements from CMake's ExternalData module.
• Easy setup and support for memory debugging which is useful when using pointers and for debugging/locating compiler issues and issues in other third party libraries or runtimes