Closed claremacrae closed 3 years ago
Confirmed - this is a real problem.
If I have a failing test due to mismatch of approved and received, the expected console output is like this:
.../MyTest.cpp:8: ERROR: test case THREW exception: Failed Approval:
Received does not match approved
Received : "out_of_source_test.out_of_source_sample.received.txt"
Approved : "out_of_source_test.out_of_source_sample.approved.txt"
If the test is run with TextDiffReporter, the output is like this:
APPROVAL_TESTS_USE_REPORTER=TextDiffReporter ./out_of_source
.../MyTest.cpp:8: ERROR: test case THREW exception: "diff" "out_of_source_test.out_of_source_sample.received.txt" "out_of_source_test.out_of_source_sample.approved.txt": failed with exit code 256
This matters because in CI builds, the default reporter is TextDiffReporter, so it makes Approval Tests failures in CI builds confusing.
This has been fixed - closing.
This needs checking, but I believe that using TextDiffReporter will change the behaviour of failing tests - from throwing the Approvals-specific mismatch exception - to throwing an exception to indicate that the “system call failed”.
This is because diff returns a non-zero exit code when files differ.
We already saw this in our own tests, but didn’t appreciate the consequences at the time.