search

catchorg / Catch2

A modern, C++-native, test framework for unit-tests, TDD and BDD - using C++14, C++17 and later (C++11 support is in v2.x branch, and C++03 on the Catch1.x branch)
https://discord.gg/4CWS9zD
Boost Software License 1.0
18.51k stars 3.04k forks source link

Fix output precision in WithinRelMatcher #2846

Closed ianhbell closed 5 months ago

ianhbell commented 6 months ago

Description

The WithinRelMatcher does not listen to the precision specification; it just naively pipes to stringstream. If you specify the precision, that has no impact on the outputted values when the match fails.

This fix makes the WithinRelMatcher listen to the precision (https://github.com/catchorg/Catch2/blob/devel/docs/tostring.md#floating-point-precision) specified by: Catch::StringMaker<float>::precision

horenmar commented 6 months ago

If this is changed to pass tests, I am willing to merge it.

ianhbell commented 6 months ago

What sort of tests do you have in mind?

horenmar commented 6 months ago

Well, right now the change does not pass CI because of the output format change. See contributing.md for how to run the tests locally.

ianhbell commented 6 months ago

I see - this breaks quite a few tests.

ianhbell commented 6 months ago

I tried to fix the tests, but the tests are failing on my machine.

I tried to run on my (Apple M1) machine: 

(base) XXX:Catch2 ihb$ ./tools/scripts/buildAndTest.sh

This fails:

[ 45%] Building CXX object tests/CMakeFiles/SelfTest.dir/SelfTest/IntrospectiveTests/RandomNumberGeneration.tests.cpp.o
/Users/ihb/Documents/Code/ianhbellCatch2/Catch2/tests/SelfTest/IntrospectiveTests/RandomNumberGeneration.tests.cpp:503:32: error: unused variable 'lowest' [-Werror,-Wunused-const-variable]
        static constexpr float lowest = -256.125f;
                               ^
/Users/ihb/Documents/Code/ianhbellCatch2/Catch2/tests/SelfTest/IntrospectiveTests/RandomNumberGeneration.tests.cpp:548:52: error: unused variable 'expected' [-Werror,-Wunused-const-variable]
    constexpr float uniform_fp_test_params<float>::expected[];
                                                   ^
/Users/ihb/Documents/Code/ianhbellCatch2/Catch2/tests/SelfTest/IntrospectiveTests/RandomNumberGeneration.tests.cpp:525:33: error: unused variable 'lowest' [-Werror,-Wunused-const-variable]
        static constexpr double lowest = -234582.9921875;
                                ^
/Users/ihb/Documents/Code/ianhbellCatch2/Catch2/tests/SelfTest/IntrospectiveTests/RandomNumberGeneration.tests.cpp:549:54: error: unused variable 'expected' [-Werror,-Wunused-const-variable]
    constexpr double uniform_fp_test_params<double>::expected[];

and then a bunch of other tests fail:

The following tests FAILED:
      1 - RunTests (Not Run)
      2 - List::Tests::Output (Not Run)
      3 - List::Tests::Quiet (Not Run)
      4 - List::Tests::ExitCode (Not Run)
      5 - List::Tests::XmlOutput (Not Run)
      6 - List::Tags::Output (Not Run)
      7 - List::Tags::ExitCode (Not Run)
      8 - List::Tags::XmlOutput (Not Run)
      9 - List::Reporters::Output (Not Run)
     10 - List::Reporters::ExitCode (Not Run)
     11 - List::Reporters::XmlOutput (Not Run)
     12 - List::Listeners::Output (Not Run)
     13 - List::Listeners::ExitCode (Not Run)
     14 - List::Listeners::XmlOutput (Not Run)
     15 - NoAssertions (Not Run)
     16 - TestSpecs::CombiningMatchingAndNonMatchingIsOk-1 (Not Run)
     17 - TestSpecs::CombiningMatchingAndNonMatchingIsOk-2 (Not Run)
     18 - TestSpecs::NoMatchedTestsFail (Not Run)
     19 - TestSpecs::OverrideFailureWithNoMatchedTests (Not Run)
     20 - TestSpecs::OverrideAllSkipFailure (Not Run)
     21 - TestSpecs::NonMatchingTestSpecIsRoundTrippable (Not Run)
     22 - Warnings::UnmatchedTestSpecIsAccepted (Not Run)
     23 - Warnings::MultipleWarningsCanBeSpecified (Not Run)
     24 - TestSpecs::WarnUnmatchedTestSpecFailsWithUnmatchedTestSpec (Not Run)
     25 - UnmatchedOutputFilter (Not Run)
     26 - FilteredSection-1 (Not Run)
     27 - FilteredSection-2 (Not Run)
     28 - FilteredSection::GeneratorsDontCauseInfiniteLoop-1 (Not Run)
     29 - FilteredSection::GeneratorsDontCauseInfiniteLoop-2 (Not Run)
     30 - ApprovalTests (Failed)
     31 - RegressionCheck-1670 (Not Run)
     32 - VersionCheck (Not Run)
     33 - LibIdentityTest (Not Run)
     34 - FilenameAsTagsTest (Not Run)
     35 - FilenameAsTagsMatching (Not Run)
     36 - EscapeSpecialCharactersInTestNames (Not Run)
     37 - NegativeSpecNoHiddenTests (Not Run)
     38 - TestsInFile::SimpleSpecs (Not Run)
     39 - TestsInFile::EscapeSpecialCharacters (Not Run)
     40 - TestsInFile::InvalidTestNames-1 (Not Run)
     41 - TagAlias (Not Run)
     42 - RandomTestOrdering (Failed)
     44 - Benchmarking::SkipBenchmarkMacros (Not Run)
     45 - Benchmarking::FailureReporting::ThrowingBenchmark (Not Run)
     46 - Benchmarking::FailureReporting::FailedAssertion (Not Run)
     47 - Benchmarking::FailureReporting::FailMacro (Not Run)
     48 - Benchmarking::FailureReporting::ShouldFailIsRespected (Not Run)
     49 - ErrorHandling::InvalidTestSpecExitsEarly (Not Run)
     50 - MultiReporter::CapturingReportersDontPropagateStdOut (Not Run)
     51 - MultiReporter::NonCapturingReportersPropagateStdout (Not Run)
     52 - Outputs::DashAsOutLocationSendsOutputToStdout (Not Run)
     53 - Reporters::DashAsLocationInReporterSpecSendsOutputToStdout (Not Run)
     54 - Reporters::ReporterSpecificColourOverridesDefaultColour (Not Run)
     55 - Reporters::UnrecognizedOptionInSpecCausesError (Not Run)
     56 - Colours::ColourModeCanBeExplicitlySetToAnsi (Not Run)
     57 - Reporters::JUnit::NamespacesAreNormalized (Not Run)
     63 - Reporters:Filters:compact (Not Run)
     64 - Reporters:RngSeed:compact (Not Run)
     65 - Reporters:Filters:console (Not Run)
     66 - Reporters:RngSeed:console (Not Run)
     67 - Reporters:Filters:JUnit (Not Run)
     68 - Reporters:RngSeed:JUnit (Not Run)
     69 - Reporters:Filters:SonarQube (Not Run)
     70 - Reporters:RngSeed:SonarQube (Not Run)
     71 - Reporters:Filters:TAP (Not Run)
     72 - Reporters:RngSeed:TAP (Not Run)
     73 - Reporters:Filters:XML (Not Run)
     74 - Reporters:RngSeed:XML (Not Run)
     75 - Reporters:Filters:JSON (Not Run)
     76 - Reporters:RngSeed:JSON (Not Run)
     77 - TestShardingIntegration (Failed)
     78 - TestSharding::OverlyLargeShardIndex (Not Run)
     79 - MinDuration::SimpleThreshold (Not Run)
     80 - MinDuration::DurationOverrideYes (Not Run)
     81 - MinDuration::DurationOverrideNo (Not Run)
     82 - CATCH_CONFIG_PREFIX_ALL (Not Run)
     83 - CATCH_CONFIG_DISABLE-1 (Not Run)
     84 - CATCH_CONFIG_DISABLE-2 (Not Run)
     85 - CATCH_CONFIG_DISABLE_EXCEPTIONS-1 (Not Run)
     86 - CATCH_CONFIG_BAZEL_REPORTER-1 (Failed)
     87 - NO_CATCH_CONFIG_BAZEL_REPORTER-1 (Failed)
     88 - BazelEnv::TESTBRIDGE_TEST_ONLY (Not Run)
     89 - BazelEnv::Sharding (Failed)
     90 - CATCH_CONFIG_DISABLE_EXCEPTIONS-2 (Not Run)
     91 - CATCH_CONFIG_DISABLE_EXCEPTIONS-3 (Not Run)
     92 - CATCH_CONFIG_DISABLE_EXCEPTIONS-4 (Not Run)
     93 - DeferredStaticChecks (Not Run)
     94 - FallbackStringifier (Not Run)
     95 - CATCH_CONFIG_DISABLE_STRINGIFICATION (Not Run)
     96 - PartialTestCaseEvents (Failed)
     97 - BenchmarksInCumulativeReporter (Not Run)
     98 - Reporters::registration-is-case-preserving (Not Run)
     99 - Reporters::selection-is-case-insensitive (Not Run)
    100 - Reporters::CapturedStdOutInEvents (Not Run)
    101 - MultiReporter::NoncapturingListenerDoesntCauseStdoutPassThrough (Not Run)
    102 - MultiReporter::CapturingListenerCausesStdoutCapture (Not Run)
    103 - Reporters::PreferencesForPassingAssertionsIsRespected (Not Run)
    104 - MultiReporter::PreferencesForPassingAssertionsIsRespected (Not Run)
    105 - ListenersGetEventsBeforeReporters (Not Run)
    106 - CustomArgumentsForReporters (Not Run)
    107 - DuplicatedTestCases::SameNameAndTags (Not Run)
    108 - DuplicatedTestCases::SameNameDifferentTags (Not Run)
    109 - DuplicatedTestCases::DuplicatedTestCaseMethods (Not Run)
    110 - DuplicatedTestCases::DuplicatedTestCaseMethodsDifferentFixtures (Not Run)
    111 - Reporters::RegistrationErrorsAreCaught (Not Run)
    112 - ReporterEvents::AssertionStartingHappensBeforeAssertionIsEvaluated (Not Run)
    113 - TestSpecs::EmptySpecWithNoTestsFails (Not Run)
    114 - TestSpecs::OverrideFailureWithEmptySpec (Not Run)
    115 - List::Listeners::WorksWithoutRegisteredListeners (Not Run)
    116 - TestSpecs::SkippingAllTestsFails (Not Run)
    117 - AmalgamatedFileTest (Not Run)
horenmar commented 5 months ago

Oh right, the warning issue in #2845.

I should have that fixed in a bit.

horenmar commented 5 months ago

Done. The build on M1 mac should no longer cause warning and thus fail due to Werror.

ianhbell commented 5 months ago

I think I have fixed the tests, waiting on the tests to run to be sure.

codecov[bot] commented 5 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.09%. Comparing base (029fe3b) to head (5a5382f).

Additional details and impacted files ```diff @@ Coverage Diff @@ ## devel #2846 +/- ## ======================================= Coverage 91.09% 91.09% ======================================= Files 197 197 Lines 8382 8382 ======================================= Hits 7635 7635 Misses 747 747 ```
ianhbell commented 5 months ago

Ready for merge I think

horenmar commented 5 months ago

Yep, thanks.