snitch-org / snitch

Lightweight C++20 testing framework.
Boost Software License 1.0
262 stars 7 forks source link

Generate a test event on successful assertion #86

Closed cschreib closed 1 year ago

cschreib commented 1 year ago

Currently, snitch will only generate a test event if an assertion fails, e.g., for CHECK(1 == 2), but not for CHECK(1 == 1). A normal test run would generate tons of success events, so they probably wouldn't be enabled all the time. But they could be useful to double-check that a test has indeed been implemented correctly and run as expected. They could also be necessary to audit the test results (e.g., print out and record all the checks that were performed).

Our infrastructure allows us to do this, and it may make assertion macros a bit more straightforward to implement. Perhaps even reduce compilation times in theory. It could come at a runtime performance though, as generating more events means more calls into the reporter, even if the reporter ends up ignoring them. Perhaps we could make the verbosity filters applied in registry, before the events are sent to the reporter?

cschreib commented 1 year ago

Work on this started in branch https://github.com/cschreib/snitch/tree/success-report. A first naive implementation shows both a compile-time and run-time cost. Compilation time is increased by +7% in Debug, and +18% in Release. Runtime in unchanged in Release, but increased by +50% in Debug (!). This isn't really acceptable, so a few optimisations will be necessary.

cschreib commented 1 year ago

I have consolidated the test macros to avoid code duplication. Now the compilation time with this branch is unaffected in Debug, and increased by only +8% in Release. Run time is still unaffected in Release, but still increased by +23% in Debug.

cschreib commented 1 year ago

In the work just above, I had moved the logic "abort vs continue" (for REQUIRE vs CHECK) inside the registry::report_* functions to simplify the test macros further. This had the unpleasant side-effect of preventing these functions from being noexcept. I reverted that bit, so that the functions are noexcept again. Now the compilation time is faster than on the main branch :partying_face: (-4% in Debug and -3% Release). It seems the compiler really likes the code in the test macros to be fully noexcept, which is the case for CHECK().

I also changed the verbosity to directly affect events being sent, so that unless verbosity is set to full, assertion successes should have very little performance impact. Indeed, the runtime overhead in Debug is now only +11%.

I'll take that!