approvals / ApprovalTests.cpp

Native ApprovalTests for C++ on Linux, Mac and Windows
https://approvaltestscpp.readthedocs.io/en/latest/
Apache License 2.0
316 stars 51 forks source link

TextDiffReporter should not throw if files differ #106

Closed claremacrae closed 3 years ago

claremacrae commented 4 years ago

This needs checking, but I believe that using TextDiffReporter will change the behaviour of failing tests - from throwing the Approvals-specific mismatch exception - to throwing an exception to indicate that the “system call failed”.

This is because diff returns a non-zero exit code when files differ.

We already saw this in our own tests, but didn’t appreciate the consequences at the time.

claremacrae commented 3 years ago

Confirmed - this is a real problem.

If I have a failing test due to mismatch of approved and received, the expected console output is like this:

.../MyTest.cpp:8: ERROR: test case THREW exception: Failed Approval: 
Received does not match approved 
Received : "out_of_source_test.out_of_source_sample.received.txt" 
Approved : "out_of_source_test.out_of_source_sample.approved.txt"

If the test is run with TextDiffReporter, the output is like this:

APPROVAL_TESTS_USE_REPORTER=TextDiffReporter ./out_of_source
.../MyTest.cpp:8: ERROR: test case THREW exception: "diff" "out_of_source_test.out_of_source_sample.received.txt" "out_of_source_test.out_of_source_sample.approved.txt": failed with exit code 256
claremacrae commented 3 years ago

This matters because in CI builds, the default reporter is TextDiffReporter, so it makes Approval Tests failures in CI builds confusing.

claremacrae commented 3 years ago

This has been fixed - closing.