boost-ext / ut

C++20 μ(micro)/Unit Testing Framework
https://boost-ext.github.io/ut
Boost Software License 1.0
1.26k stars 120 forks source link

Number of asserts is reported wrongly #624

Open SimonMarynissen opened 6 months ago

SimonMarynissen commented 6 months ago

For the following very simple code:

#include <boost/ut.hpp>
#include <vector>

int main() {
  using namespace boost::ut;

  "args"_test = [](const auto& arg) {
    expect(arg > 0_i) << "all values greater than 0";
  } | std::vector{1, 2, 3};
}

Expected Behavior

It outputs Suite 'global': all tests passed (6 asserts in 3 tests)

Actual Behavior

It should output Suite 'global': all tests passed (3 asserts in 3 tests)

It is not always just double the number of asserts. For example on my own code I got Suite 'global': all tests passed (1806 asserts in 84 tests), while each test has a single expect.

Steps to Reproduce the Problem

Check out on compiler explorer.

Specifications

DNKpp commented 4 months ago
From a quick view, I would guess that it somehow sums up the previous assertions twice + the current assertion. To be more precise, I populated the test vector with different elements: Elements Reported Assertions Reported tests
1 1 1
1, 3 3 2
1, 3, 4 6 3
1, 3, 4, 5 10 4

So it seems it sums up the partial sum of processed assertions (1 + 2 + 3 + 4 for the last case; like a gauss summation) instead of the actual number of assertions. You get the point. Adding more assertions multiplies the number of assertions.

But that doesn't hold with your observation of getting 1806 assertions in 84 test...

EDIT: Have a look here. It's getting clearer when the expectation fails: https://godbolt.org/z/njea4c9sx The output actually shows duplicated assertions (and warns about them).