Make sure we actually run tests at runtime

tcbrindle commented 1 year ago

Because almost the entire library is constexpr, the majority of our tests look like this:

constexpr bool test_xxx() {
    // test code
    return true;
}
static_assert(test_xxx());

TEST_CASE("xxx") {
    REQUIRE(test_xxx());
}

That is, we do a constexpr test during compilation (which has the added benefit of testing for many guises of UB), and then again at runtime using Catch.

Unfortunately, compilers are getting increasingly smart at noticing that they've already calculated the result of test_xxx(), so they can optimise the run-time version to just REQUIRE(true), which isn't exactly in the spirit of what we want as it means we never actually execute the tests at runtime! It also means that the tested functions never show up in our code coverage measurements.

As a workaround, this commit adds an otherwise-useless template <bool = true> to every test_xxx() function. Then in the runtime version we explicitly call test_xxx<false>() to force a separate instantation which should hopefully show up in CodeCov.

This probably won't do wonders for our compile times, but we'll see...

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage has no change and project coverage change: +0.01 :tada:

Comparison is base (471dd38) 97.70% compared to head (fa91d33) 97.71%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #70 +/- ## ========================================== + Coverage 97.70% 97.71% +0.01% ========================================== Files 62 62 Lines 2045 2057 +12 ========================================== + Hits 1998 2010 +12 Misses 47 47 ``` [see 1 file with indirect coverage changes](https://app.codecov.io/gh/tcbrindle/flux/pull/70/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Tristan+Brindle)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

tcbrindle commented 1 year ago

Well, that was a complete waste of time.

It looks like GCC is perfectly capable of executing test_xxx<false>() at compile-time and using that, even though we didn't explicitly ask for it.

Ah well, never mind.

tcbrindle / flux

Make sure we actually run tests at runtime #70

Codecov Report