Error regular expression found in output in Failing Tests

[Brian-Waite-nnl]

Hello,

I am currently having an issue where a failing test always reports the error:

Error regular expression found in output. Regex=[Encountered 1 or more failures/errors during testing]

Have you seen this issue before?

I am running linux with gfortran 9.3 but I have reproduced it with an intel compiler as well.

If I take the / out of the string on line 171 in FUnit.f90 in src/funit -- this error goes away. However, when I did this, I needed to add the error stop back instead of just stop to get it to catch the error properly and report the failed test. (This brings the backtrace again which a recent commit avoided). Otherwise tests were shown as passing even though they failed which may mean the the error code was coming from the presence of the backslash with just the stop command?

For instance in funit.f90:

171: error stop '*** Encountered 1 or more failures or errors during testing. ***'

instead of

171: stop '*** Encountered 1 or more failures/errors during testing. ***'

However, this may be more fixing the symptom than the cause so I thought I would reach out.

Thanks!

[Brian-Waite-nnl]

Hello! I am coming back to this am wondering if this is a system issue with me or if others have seen this.

Further details using the trivial tests in the examples repository and changing it to intentionally fail and running ctest -V.

1: Test timeout computed to be: 10000000
1: .F
1: Time:         0.000 seconds
1:   
1: Failure
1:  in: 
1: test_square_suite.test_square
1:   Location: 
1: [test_square.pf:6]
1: does square(3) work
1: AssertEqual failure:
1:       Expected: <9.10000038>
1:         Actual: <9.00000000>
1:     Difference: <-0.100000381> (greater than tolerance of 0.00000000)
1:   
1:  FAILURES!!!
1: Tests run: 1, Failures: 1, Errors: 0
1: , Disabled: 0
1: STOP *** Encountered 1 or more failures/errors during testing. ***
1/1 Test #1: my_tests .........................***Failed  Error regular expression found in output. Regex=[Encountered 1 or more failures/errors during testing]  0.00 sec

If I remove the "/" in src/funit/FUnit.f90 it shows test passes but does show the test failure as shown below.

1: Test timeout computed to be: 10000000
1: .F
1: Time:         0.000 seconds
1:   
1: Failure
1:  in: 
1: test_square_suite.test_square
1:   Location: 
1: [test_square.pf:6]
1: does square(3) work
1: AssertEqual failure:
1:       Expected: <9.10000038>
1:         Actual: <9.00000000>
1:     Difference: <-0.100000381> (greater than tolerance of 0.00000000)
1:   
1:  FAILURES!!!
1: Tests run: 1, Failures: 1, Errors: 0
1: , Disabled: 0
1: STOP *** Encountered 1 or more failures or errors during testing. ***
1/1 Test #1: my_tests .........................   Passed    0.00 sec

Maybe the "/" is intentional to catch the error but it seems strange to me that it depends on that.

any info would be helpful!

[tclune]

First - apologies that I did not respond in the first go around. I must have missed it or forgot to respond when I got distracted.

The string in the Fortran source code is the string that the CMake/CTest macro is checking to see if the test succeeded or failed. As far as FUnit proper is concerned, the test was failing either way, but CMake needs to have some simple definition of failure. The "easy" way is to look for a non-zero result.

Fortran does not define a result values for executables, but sensible implementations give a non-zero result with error stop. And as you've noted, GFortran is a bit unfriendly these days with error stop because it gives a stack trace. (In parallel is is very annoying.

So the work around was to emit a specific string and have CMake look for that string: https://github.com/Goddard-Fortran-Ecosystem/pFUnit/blob/26ac761630dd3012081d67ce03ee99bff2dbe8d9/include/add_pfunit_ctest.cmake#L140-L142

I'm not wed to the particular string - we just need the two to agree on what is a failure. And it ought to be specific enough that a random print statement would not accidentally match.

[Brian-Waite-nnl]

No worries on not responding right away, it was around the holidays and it was only a minor question. Sounds like I was a little bit wrapped around the axle on this one. I think the expectation a failed test would simply say "**failed" so the extra output around the regular expression. This makes sense, thanks very much for taking the time to clear it up for me.