Equivalent to Doctest's SUBCASE and CHECK_NOTHROW

[nlohmann]

I would like to give snitch a try, but my current Doctest test suite heavily relies on SUBCASE and CHECK_NOTHROW. Are there plans to integrate these to snitch as well?

[cschreib]

Hi, and thank you for your interest!

• SUBCASE is already implemented, it is called SECTION (following the Catch2 API).
• CHECK_NOTHROW is not currently available in snitch, but can be added without much effort. Until I get around to merge and release this, you can add it yourself using snitch v1.1.1 by simply defining these macros in your code:

  
  #define SNITCH_REQUIRE_NOTHROW_IMPL(MAYBE_ABORT, ...)                                              \
      do {                                                                                       \
          auto& SNITCH_CURRENT_TEST = snitch::impl::get_current_test();                          \
          try {                                                                                  \
              static_cast<void>(__VA_ARGS__);                                                    \
              SNITCH_CURRENT_TEST.reg.report_assertion(                                          \
                  true, SNITCH_CURRENT_TEST, {__FILE__, __LINE__},                               \
                  #__VA_ARGS__ " did not throw");                                                \
          } catch (...) {                                                                        \
              try {                                                                              \
                  throw;                                                                         \
              } catch (const std::exception& e) {                                                \
                  SNITCH_CURRENT_TEST.reg.report_assertion(                                      \
                      false, SNITCH_CURRENT_TEST, {__FILE__, __LINE__},                          \
                      "expected " #__VA_ARGS__                                                   \
                      " not to throw but it threw a std::exception; message: ",                  \
                      e.what());                                                                 \
              } catch (...) {                                                                    \
                  SNITCH_CURRENT_TEST.reg.report_assertion(                                      \
                      false, SNITCH_CURRENT_TEST, {__FILE__, __LINE__},                          \
                      "expected " #__VA_ARGS__                                                   \
                      " not to throw but it threw an unknown exception");                        \
              }                                                                                  \
              MAYBE_ABORT;                                                                       \
          }                                                                                      \
      } while (0)

define SNITCH_REQUIRE_NOTHROW(...) SNITCH_REQUIRE_NOTHROW_IMPL(SNITCH_TESTING_ABORT, __VA_ARGS__)

define SNITCH_CHECK_NOTHROW(...) SNITCH_REQUIRE_NOTHROW_IMPL((void)0, __VA_ARGS__)

if SNITCH_WITH_SHORTHAND_MACROS

define REQUIRE_NOTHROW(...) SNITCH_REQUIRE_NOTHROW(__VA_ARGS__)

define CHECK_NOTHROW(...) SNITCH_CHECK_NOTHROW(__VA_ARGS__)

endif

If you do get around to testing *snitch*, I'd be very interested in hearing any feedback, even (especially?) if negative.

[nlohmann]

Thanks for the incredibly quick response!

I checked a bit more, and the test suite currently relies on further features that are not yet realized (and some seem not on your roadmap):

• Approx (we do some math, and float comparison sucks...)
• CHECK_THROWS_WITH_AS - I did not manage to map this to CHECK_THROWS_MATCHES
• WARN - not a deal breaker, but convenient when tests rely on environment variables and you just don't care if they fail on some systems

I removed these tests and got the code to compile. :-)

Some observations:

• I like the output of failing tests. This is really helpful. Unfortunately, the tests do not integrate as nicely into CLion as Doctest or Catch do. This is unfortunate, because this integration makes it very easy to repeat failing tests or to just execute a single test in the debugger.

• There seem to lack an option to generate JUnit XML output. We use this in our CI (Gitlab) to collect reports.

[cschreib]

Thanks a lot for the feedback! I'll respond to each point below.

Approx (we do some math, and float comparison sucks...)

Oddly enough, although it was never brought up before, you are now the second person to mention floating point matchers today. I haven't used these matchers myself, but it was pointed out to me that Approx is considered deprecated in Catch2, and that one should use the new floating point matchers instead. Is this something you have considered?

As I mentioned to the other person (on reddit), if I am to implement floating point matchers, I would like to understand what people actually use, but also what they actually need (which may not be the same thing). Although I can browse other libraries' source code to figure out the former, I need to actually talk to a human for the latter :) So if you are happy to share more information about the tests you perform, and what it is you are after, I'd love to hear it.

CHECK_THROWS_WITH_AS

Here is a substitute that should work (assuming your exception type implements the standard what() member function to get the error message; if not, you will need a different matcher):

#define CHECK_THROWS_WITH_AS(EXPR, MESSAGE, EXCEPTION) \
    CHECK_THROWS_MATCHES(EXPR, EXCEPTION, snitch::matchers::with_what_contains{MESSAGE})

WARN - not a deal breaker, but convenient when tests rely on environment variables and you just don't care if they fail on some systems

I haven't been able to find an occurrence of WARN in https://github.com/nlohmann/json, but based on what you said above, it seems to me that this could be replaced by the special tag [!mayfail]:

TEST_CASE("platform dependent stuff", "[!mayfail]") {
    CHECK(some_platform_dependent_things());
}

Any failure in a test case tagged with [!mayfail] will be reported (e.g., with the default reporter, it will appear as a failure in the standard output), but the test case will always be considered as "passed":

starting snitch_runtime_tests_self with snitch v1.1.1.5ad2fff
==========================================
failed: running test case "platform dependent stuff"
          at test.cpp:5
          CHECK(some_platform_dependent_things()), got false
==========================================
success: all tests passed (1 test cases, 1 assertions)

• I like the output of failing tests. This is really helpful. Unfortunately, the tests do not integrate as nicely into CLion as Doctest or Catch do. This is unfortunate, because this integration makes it very easy to repeat failing tests or to just execute a single test in the debugger.
• There seem to lack an option to generate JUnit XML output. We use this in our CI (Gitlab) to collect reports.

I believe the two are related. snitch currently only implements a human-readable reporter (the default), and an example for TeamCity messages. We are able to handle machine-readable XML output, it just requires writing a reporter for it. This has been requested before, and I would like to include this. However I am unsure which format would be most useful. It seems Catch2 implements both a JUnit XML reporter and its own XML reporter. Same for doctest.

Rather than map to a standard interface, it seems IDEs implement specific "test adapters" for each testing framework. I suppose this is required anyway for handling stuff like command line inputs for selecting a particular test... snitch has its own command line interface, but it is very close to the one implemented in Catch2. I'll have to see if we can just piggy-back on the existing Catch2 test adapters, and have them magically work with snitch as well.

[nlohmann]

• I have not yet looked at anything other than doctest::Approx. I can imagine other APIs, but basically I just want to express "these two numbers must be the same after rounding them up to x digits" or "the difference between these two numbers must not exceed x". My concrete use case involve digital maps, and the tests check the coordinates of intersection points and distances. Both are (unfortunately), not 100% same on all systems, but the domain allows some impreciseness (see https://xkcd.com/2170/).
• I did not try snitch with nlohmann/json, but with a project I develop at my day job. There, WARN is used for a specific test where environment variables control a logging level, and a test is querying the default logging level. In the CI, we have fixed the environment, so the result is predictable. But if your local setup deviates, this must not make the test suite fail - a warning is sufficient. I can try [!mayfail].
• Thanks for the definition of CHECK_THROWS_WITH_AS. I'll give it a try.
• I cannot comment on your roadmap, but I think making snitch behave (a bit) like Catch2 could save you quite some work that the community did in the past - be it the integration to IDEs or the command-line arguments. I'm afraid there is too little room for originality here that it could be easier to just stick with a de-facto standard.
• We use GitLab CI which can collect test reports in JUnit format. It allows you to see the results right in merge request without the need to create/download fancy HTML reports or stare at console outputs. Again, using a standard can help to avoid inventing something new.

That said, I will check the latest updates and report back to you how they work.

[nlohmann]

FYI: It seems CLion tries to call Doctest suites with -r=xml - maybe (?) this would be all for an integration.

[cschreib]

Indeed Catch2 has a similar flag; --reporter xml. I think this is what we'll aim for, and will track this in a separate issue: https://github.com/cschreib/snitch/issues/100. I also opened a separate issue to track the implementation of the float matchers: https://github.com/cschreib/snitch/issues/101.

Thank you for all the feedback! I will close this issue as a resolved question, and if you're interested in following up the discussion on the other topics, feel free to participate there ;)