with CppUTest: `ApprovalMismatchException` suppresses other tests output

[nitsanavni]

see example here

the example shows a test file that has two failing tests - one of them is using approvals, the other is a non-approvals test.

expected

both failures should be shown

actual

the non-approvals test output is not seen in the terminal

[claremacrae]

Things we found when experimenting with this today:

• CppUTest executes test from the end of the file, in reverse order
• As soon as an exception is thrown in any source file, execution in that file halts

I wonder if this long discussion on a merged CppUTest PR provides any hints on how to work around this: https://github.com/cpputest/cpputest/pull/1550

[claremacrae]

I wonder if this long discussion on a merged CppUTest PR provides any hints on how to work around this: cpputest/cpputest#1550

Yes, it did. Adding either of these two command-line arguments when running the tests makes both test failures be shown:

-e               - do not rethrow unexpected exceptions on failure
-ci              - continuous integration mode (equivalent to -e)

Confirmed with this change:

git diff
diff --git a/Makefile b/Makefile
index 86536c0..fd89508 100644
--- a/Makefile
+++ b/Makefile
@@ -21,7 +21,7 @@ clean :
        git clean -fx

 test: ExampleTestAndApprovalTest
-       ./ExampleTestAndApprovalTest
+       ./ExampleTestAndApprovalTest -e

 exampleonly: OnlyExampleTest
        ./OnlyExampleTest

The console output is then:

gitpod /workspace/approvals-cpp-exception (main) $ make test
./ExampleTestAndApprovalTest -e

./ExampleTestAndApprovalTest.cc:31: error: Failure in TEST(TestGroup, ApprovalTestsTest)
        Unexpected exception of type 'ApprovalTests::ApprovalMismatchException' was thrown: Failed Approval: 
Received does not match approved 
Received : "./ExampleTestAndApprovalTest.TestGroup.ApprovalTestsTest.received.txt" 
Approved : "./ExampleTestAndApprovalTest.TestGroup.ApprovalTestsTest.approved.txt"

.
./ExampleTestAndApprovalTest.cc:28: error: Failure in TEST(TestGroup, ExampleBasedTest)
        expected <a>
        but was  <b>
        difference starts at position 0 at: <          b         >
                                                       ^

.
Errors (2 failures, 2 tests, 2 ran, 1 checks, 0 ignored, 0 filtered out, 3 ms)

make: *** [Makefile:24: test] Error 2
gitpod /workspace/approvals-cpp-exception (main) $ 

[claremacrae]

We should probably add this to the documentation on the Approval Tests/CppUTest integration.

[nitsanavni]

super!

thanks so much for taking the time!