provide Python script to run functional tests and compare output files

[jwuttke]

This is a prerequisite for #23.

Proposed name of the script: run_steca_ftest.

Proposed language: Python3. Enhance ftest/CMakeLists.txt to make this language available. Alternative: stay with C++.

The script shall take 4 arguments: <Steca> <command_file> <out_dir> <ref_dir> <timeout> <tolerance> where

• <Steca> is the Steca executable
• <command_file> is made of Steca capture&replay commands, and controls one Steca session
• <out_dir> is the directory where Steca, according to <command_file>, will write its output files
• <ref_dir> is the reference directorythat holds exact copies of the expected output files
• <timeout> is the maximum allowed execution time in seconds
• <tolerance> is a number (typically 1e-4) that defines a relative tolerance for deviations of floating-point outcomes.

All file or directory parameters may be absolute or relative paths.

The script shall do the following:

• Create <out_dir>, or empty it if it exists.
• Run the shell command <Steca> [options] <command_file> where [options] are to be specified later. If Steca fails, then terminate (return 1).
• Enhancement of the above: run Steca in a thread that is forcibly terminated if execution takes longer than <timeout>; in that case, terminate the script (return 9).
• Assert that <out_dir> and <ref_dir> contain the same number of files, with identical file names. Otherwise terminate (return 2).
• Compare pairs of files (one from each directory). Break files at whitespaces into "words". If any pair of words differs, then print a list of differences, and terminate (return 3).
• From the previous rule, except however words that are floating-point numbers. Convert them from string to number, and compare them numerically. If any pair differs by more than the allowed relative tolerance, than print a list of pairs that do so, and terminate (return 4).
• Otherwise, terminate successfully (return 0).

[jwuttke]

Resolved in part by the script 'numdiff' written by Janike Katter.

[jwuttke]

transferred to jugit