How to add custom output on test failure (for binary data comparison)

[Malvineous]

Description

I would like to port my tests away from Boost.Test and I have started investigating Catch. Everything looks good except I can't work out how to cleanly compare binary data with Catch. Because my code produces binary files, if a test fails because the output data is not what was expected, I need to see the offending data in the test output so I can work out what went wrong.

Steps to reproduce

TEST_CASE("Binary data comparison") {
    std::string a( "test\x00\x01\x02\x03ing!", 12 );
    std::string b( "test\x03\x02\x01\x00ing!", 12 );

    REQUIRE( a == b );
}

Output:

tests_catch.cpp:69: FAILED:
  REQUIRE( a == b )
with expansion:
  "testing!" == "testing!"

The binary data is not present in this format, and if it were, it would not be as clear as it could be.

Extra information

Boost.Test lets me write a custom output function, so I wrote one that looks like this, using ANSI codes to colour the differing bytes:

test.cpp(566): error: in "test_xyz/test_open[xyz]": Example failure. 
Exp: 74 65 73 74 00 01 02 03 69 6e 67 21              test....ing!
Got: 74 65 73 74 03 02 01 00 69 6e 67 21              test....ing!

Is it possible to do the same with Catch?

[horenmar]

I get a slightly different output, but the answer is that it depends.

First of all, you can't get the desired output without a little bit of hacking, because normally each side of the comparison is stringified on its own, so even if you use your own type instead of std::string, the output wouldn't be able to have colouring dependent on the mismatch.

The closest you can get to it is to have your own function that does the comparison and uses INFO to provide formatted output. This will give you the least support from Catch, but has the most freedom.

void my_test(binary_string const& lhs, binary_string const& rhs) {
    if (lhs == rhs) {
        SUCCEED();
    }
    std::stringstream sstream;
    // Now format the output as you want

    INFO(sstream.str());
    FAIL();
}

It can then be used in a test just by calling it:

TEST_CASE("a test case", "[foo]") {
    binary_string left, right;
    my_test(left, right);
}

Another option is to use Matchers. Matchers allow you to have fairly arbitrary logic and stringification, up to a point -- you still cannot change how the matcher's argument is stringified, but you can change how the matcher itself is stringified, because it is only stringified after a matching attempt.

The usage and output could look something like

CHECK_THAT(left, BinaryMatcher(right));

74 65 73 74 00 01 02 03 69 6e 67 21 does not match
74 65 73 74 *color*03 02 01 00*end color* 69 6e 67 21

this also assumes that you have type different from std::string, because std::string's stringification is hard-wired to be done as characters (give or take escaping some special ones). Otherwise the left side would be stringified as a string with unprintable characters (weirdly).

[Malvineous]

Many thanks for the details! I've been investigating this and it looks like it might work, except the INFO() call seems to perform very short line wrapping:

tests_catch.cpp:124
...............................................................................

tests_catch.cpp:71: FAILED:
explicitly with message:
  74 65 73 74 00 01 02 03 69 6e 67 21     test
....ing!

Is there any way to avoid having it wrap? I'm guessing it's because it is counting the ANSI colour codes as printable characters and thus wrapping the line too soon, but sometimes I like to have the lines really long temporarily (e.g. 200 chars wide) because it can make comparing the expected vs actual values much easier. So rather than "fixing" the line wrapping, in this case it would be more useful if I could turn it off, just for those messages I am generating with INFO().

Is that a possibility? Thanks again!

[horenmar]

Sorry for not responding for so long, it kinda fell through the cracks.

Anyway, as far as I know there is no way of temporarily turning off formatting for part of the output. However, depending on how much you use the line wrapping otherwise, you might get good value out of turning it off completely. To turn it off completely, define CATCH_CONFIG_CONSOLE_WIDTH to some ludicrous value and Catch will stop trying breaking lines on its own (well, it will try to break on whatever the value of CATCH_CONFIG_CONSOLE_WIDTH is).

[jewalker]

I also have a similar question. How would you achieve a similar result using Matchers (assuming the data was something other than a string)? Looking at the definition for a Matcher the match() function is declared as const, but the describe() function for returning the error reporting string is called outside of match, so how do you pass the necessary information discovered in match() to describe()? Say the data under test was megabytes long and you wanted to point out that bytes 345-348 differed, and maybe print out the differing data. What's the best way to do that?

[jewalker]

I think I just answered this question myself by looking deeper into the header file. The Matcher has a function toString() which by default checks to see if the member variable m_cachedToString is empty. If not then it returns this variable, otherwise it returns the result of describe(). m_cachedToString is marked as mutable allowing it to be modified in the const match() function (which is how I was considering achieving it). It would be very useful if you could add this information and possibly an example to the documentation for Matchers. As it stands now the documentation states that you must override describe() and makes no mention of m_cachedToString.

[philsquared]

Unfortunately that's not how it is supposed to work. Calls to describe() should be independent of any calls to match(). That caching string is an internal implementation detail (to avoid repeated calls to a potentially expensive stringstream operation in describe()).

describe() can depend on the value being matched against (the "right-hand-side" value) - but not the "left-hand-side" value.

To support something like this, a potential possibility is to add an optional virtual that works like describe(), but takes the lhs value for use in constructing the string.

I'll have to give that some thought...

[jewalker]

The information needed to construct the error message is available immediately in match(), so it would be nice to use that information when constructing the string. If you add an additional virtual method similar to describe() I wouldn't want to have to recreate and rerun the logic I used in match() to identify the problem just to construct the error string.

The way Boost Test handles this is that their equivalent to match() returns a struct which contains the boolean result of the test and an optional error string.

The workaround I'm using for now is to do what toString() already does: add a mutable string member variable and have describe() return that variable. It appears to work for my test cases, but you might be aware of places this breaks down.

[viridia]

I've been running into similar issues. The current Matcher API doesn't allow for detailed reports of failures, because the 'actual' and 'expected' values are treated independently.

Most assertion frameworks that I am familiar with (GoogleTest, Jest, Truth, certainty, etc.) have a way to give additional information when comparing complex entities in matchers. If I am comparing abstract syntax trees, DOM nodes, protocol buffers, or paragraphs of text, it's a burden for the programmer to have to visually diff the 'expected' value with the 'actual' value to see exactly where the divergence is.

A custom matcher should have a way to report a failure that includes both the expected and the actual value, so that it can intelligently diff the two values and provide a friendly, human-readable summary of exactly what failed. I'm perfectly willing to write a diff algorithm for my data structures, but I need a place to plug it in.

[BurningEnlightenment]

Since this is still unresolved in v3 and it looks like it won't change anytime soon, I bit the bullet and wrote a match expression and some associated assertion macros. It isn't as fancy as the picture shown in the OP -- it just highlights the first mismatch. The implementation uses some C++20 features and fmt (porting to C++14 shouldn't be too much trouble, but I don't need it). BSL-1.0 license.

blob_matcher.hpp Synopsis

namespace dp_tests
{
template <typename R>
concept blob_like
    = std::ranges::input_range<R> && std::ranges::forward_range<R>
    && (std::is_same_v<std::ranges::range_value_t<R>, std::uint8_t>
        || std::is_same_v<std::ranges::range_value_t<R>, std::byte>
        || std::is_same_v<std::ranges::range_value_t<R>, char>);

// For brevity, I have included the argument constraints as part of the macro signature
#define CHECK_BLOB_EQ(blob_like auto const &arg, blob_like auto const &expected)
#define REQUIRE_BLOB_EQ(blob_like auto const &arg, blob_like auto const &expected)
}

Example Output

C:\devel\source\deeplex\deeppack\src\dplx\dp\items\emit_ranges.test.cpp(57): FAILED:
  CHECK_BLOB_EQ( outputStream.written(), sample.encoded_bytes() )
with expansion:
  pos:     vv 0x0001
  val:    /02                                           ..
       81<
  exp:    \01                                           ..

[...]

C:\devel\source\deeplex\deeppack\src\dplx\dp\items\emit_ranges.test.cpp(57): FAILED:
  CHECK_BLOB_EQ( outputStream.written(), sample.encoded_bytes() )
with expansion:
  pos:              vv 0x000a
  val:             /c0 18 71 18 38 18 9b 18 92 0a 18 7b  ......q.8......{
       08 18 b5 18<
  exp:             \c1 18 71 18 38 18 9b 18 92 0a 18 7b  ......q.8......{

[Char-Aznable]

In GoolgeTest, there is the "result_listener" to customize the output message upon a matcher returning false. I wonder if Catch can do something similar