Cantera / cantera

Chemical kinetics, thermodynamics, and transport tool suite
https://cantera.org
Other
580 stars 341 forks source link

Failures of compiled tests not indicated in GitHub Actions results #1712

Open speth opened 1 week ago

speth commented 1 week ago

Problem description

Currently, the Fortran test suite is failing some test comparisons, but this is not indicated in GitHub Actions.

This is caused by the changes in #1703, where we no longer run the scons test meta-target, which causes SCons to exit with a non-zero status if any test fails. The more specific test failures don't automatically cause SCons to abort, which is preferable so you can collect multiple test failures in one run, but gets in the way here.

Behavior

On builders that are able to compile the Fortran module, the build step "Run compiled tests" ends with output like:

Found differences between /home/runner/work/cantera/cantera/test_problems/fortran/f77_ctlib_blessed.txt and /home/runner/work/cantera/cantera/test_problems/fortran/f77_ctlib_output.txt:
>>>
--- 

+++ 

@@ -1,3 +1,4 @@

+DeprecationWarning: FactoryBase::canonicalize: Model name '' is deprecated. Use 'none' instead.
            1  0.10000000149011612        12043607.446437676
            2  0.10000000149011612       -30030146.685939986
            3  0.10000000149011612        1845891.1774966023
@@ -6,6 +7,6 @@

            6  0.10000000149011612        60990480.708799988
            7  0.10000000149011612        34041404.588181324
            8  0.10000000149011612       -64773610.650808521
-           9  0.10000000149011612        0.0000000000000000
+          10  0.10000000149011612        0.0000000000000000
           10  0.10000000149011612        0.0000000000000000

<<<
FAILED
Command execution time: test_problems/fortran/.passed-f77-ctlib: 0.011001 seconds
* Running test 'f77-demo'...
Comparing 'f77_demo_blessed.txt' with 'f77_demo_output.txt'
PASSED
Command execution time: test_problems/fortran/.passed-f77-demo: 0.012251 seconds
Command execution time: test-legacy: 0.000003 seconds
scons: done building targets.
Total build time: 13.485598 seconds
Total SConscript file execution time: 1.626395 seconds
Total SCons execution time: 0.918103 seconds
Total command execution time: 10.941100 seconds

and the build step is marked as completing successfully.