Boost.UT tests spuriously pass, by returning 0 exit status even after failure

[claremacrae]

When I hack the approval files in tests/UT_Tests/ApprovalTestTests.cpp to make the test fail, and then run the test via this:

ctest .

I expect to see test saying the test failed.

However, it gives

      Start  4: UT_Tests
 4/13 Test  #4: UT_Tests ................................   Passed    0.23 sec

then Test reports the tests pass, even though a diff tool popped up to say an error occurred.

ctest . --verbose

then I do see the console output, but the test still reports it passes.

      Start  4: UT_Tests

4: Test command: /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp/cmake-build-debug-g-9-brew/tests/UT_Tests/UT_Tests
4: Test timeout computed to be: 10000000
4: Running "ItCanVerifyAFile"...PASSED
4: Running "AnotherWayItCanVerifyAFile"...PASSED
4: Running "ItCanUseMultipleVerify"...
4:   Unexpected exception with message:
4: Failed Approval: 
4: Received does not match approved 
4: Received : "/Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp/tests/UT_Tests/approval_tests/ApprovalTestTests.ItCanUseMultipleVerify.section_1.received.txt" 
4: Approved : "/Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp/tests/UT_Tests/approval_tests/ApprovalTestTests.ItCanUseMultipleVerify.section_1.approved.txt"
4: FAILED
4: Running "YouCanUseAWriter"...PASSED
4: Running "YouCanSpecifyYourFileExtension"...PASSED
4: Running "YouCanSpecifyYourFileExtensionWithToString"...PASSED
4: Running "YouCanSpecifyYourFileExtensionWithFormatter"...
4:   Unexpected exception with message:
4: Failed Approval: 
4: Received does not match approved 
4: Received : "/Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp/tests/UT_Tests/approval_tests/ApprovalTestTests.YouCanSpecifyYourFileExtensionWithFormatter.received.md" 
4: Approved : "/Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp/tests/UT_Tests/approval_tests/ApprovalTestTests.YouCanSpecifyYourFileExtensionWithFormatter.approved.md"
4: FAILED
4: 
4: ===============================================================================
4: tests:   7 | 2 failed
4: asserts: 0 | 0 passed | 0 failed
4: 
 4/13 Test  #4: UT_Tests ................................   Passed    0.02 sec

I presume that this is because Boost.UT is returning a 0 exit status even after an error has occurred.

This is most serious in the case where the user's build is not set up correctly to work with Approval Tests, e.g. when using Ninja with an in-source build...

Approval Tests will throw one exception per test case, which 'ctest .' will swallow - and then ctest will report that all the tests succeeded. When in fact they all failed...

[claremacrae]

@kris-jusiak Hello, please could you advise on how to track this down? It's means that all our uses of Boost.ut spuriously pass in test, which is rather worrying.

It occurred with 20bf64b6f6a8b9bd1e87de243460b26b837ded55 in this project, which contains 8004290d195b08eda531eb015ddcda2e11d41ce9 of Boost.UT.

The source code of the test in this version is: https://github.com/approvals/ApprovalTests.cpp/blob/20bf64b6f6a8b9bd1e87de243460b26b837ded55/tests/UT_Tests/ApprovalTestTests.cpp

I thought maybe it would detect a failure on the last test case, so I changed the content of the file tests/UT_Tests/approval_tests/ApprovalTestTests.YouCanSpecifyYourFileExtensionWithFormatter.approved.md but the exit code was still 0.

[claremacrae]

@kris-jusiak 💡 I think I understand the problem. In the above two commits, I added some tests that use expect() to report any issues. And if they fail, the exit status is 255.

Approval Tests report failures by throwing exceptions, which they then assume will result in a non-zero exit status. It looks to me like if the only errors are unexpected exceptions, this doesn't get reported in Boost.UT's exit status....

Is this enough information? Thanks in advance for any help that you can offer...

[claremacrae]

This problem matters when I run Boost.UT v1.1.6 on Mac... It looks like all ApprovalTests Boost.UT verifications pass, when running the tests with ctest ., as the exit status is 0, and no messages are written out.

If ctest is run with --verbose then there is console output that indicates the failure to get the filename.

--output-on-error doesn't help, if the only failing tests are exceptions, as thrown by ApprovalTests - because the test exits with 0 (success) and so ctest thinks that everyone was all good.

I think this may have slightly contributed to the challenges understanding the Ninja problems in #74:

• it was said that v1.1.6 worked fine with Ninja...
• and if the tests were run with ctest, it certainly looked like it.
• In fact, v1.1.6 is broken on Mac for a different reason (https://github.com/boost-experimental/ut/issues/291)
• But the errors from that were only shown if the tests were run in an IDE or via the exe directly, and not via ctest

Various combinations of Boost.ut version, compilers, and running either via ctest or directly, resulted in some confusion.

@jwillikers and @lp55 please note:

• Boost.UT 8004290d195b08eda531eb015ddcda2e11d41ce9 works with Approval Tests on Mac.
• v1.1.6 does not find filenames on Mac
• and don't depend on the Boost.UT exit status from the tests to indicate ApprovalTests error.
• probably always use ctest --verbose for now.

[jwillikers]

@claremacrae Noted. Thank you for the update Clare!

[claremacrae]

I’ve got a fix for this. I’ll submit a PR to Boost.UT later this week.