catchorg / Catch2

A modern, C++-native, test framework for unit-tests, TDD and BDD - using C++14, C++17 and later (C++11 support is in v2.x branch, and C++03 on the Catch1.x branch)
https://discord.gg/4CWS9zD
Boost Software License 1.0
18.46k stars 3.02k forks source link

Attempts to Generate Code Coverage Lead to a Compilation Freeze #1253

Closed arnavb closed 5 years ago

arnavb commented 6 years ago

Description

Ok, so this is a problem I have been having for a while, but I have no way to solve. In my repository (arnavb/cpp14-project-template#1), I have been trying to add code coverage so I can see the effectiveness of my tests. To do so, I have been using this module. In order to apply code coverage, I added the following CMake code:

if (ENABLE_COVERAGE AND NOT CMAKE_CONFIGURATION_TYPES)
    if (NOT BUILD_TESTS)
        message(FATAL_ERROR "Tests must be enabled for code coverage!")
    endif (NOT BUILD_TESTS)

    include(CodeCoverage)
    append_coverage_compiler_flags()

    set(COVERAGE_EXCLUDES "${CMAKE_SOURCE_DIR}/test/*")
    setup_target_for_coverage(NAME coverage EXECUTABLE ctest DEPENDENCIES coverage)
endif (ENABLE_COVERAGE AND NOT CMAKE_CONFIGURATION_TYPES)

Now this does work, I can confirm (without using Catch), as you can see in arnavb/test-codecov in an older commit. However, when I add Catch tests to the project (it's newest commit), I get a compilation freeze since the Coverage flags are seriously slowing down the Catch code. (I waited a few minutes, and the compilation still hasn't finished). Now I thought: Hey! maybe I can just apply the flags to the library I'm testing! In order to do that, I commented out append_coverage_compiler_flags() and added the following to my src/CMakeLists.txt:

if (ENABLE_COVERAGE) # Uncomment these lines
    target_compile_options(test-codecov-lib PRIVATE "-g;-O0;--coverage;-fprofile-arcs;-ftest-coverage")
endif (ENABLE_COVERAGE)

Now this does work, but only partially. Now, when I try to compile, the compilation finishes quickly for the tests. However, if there is a function present that is defined just in a header file (such as a template function), it's coverage is completely ignored! I asked a question on SO to try and solve this and got the following response in a comment:

So I need to apply flags to the header files? - There is no such thing like "compiler flags for a header file" - compiler flags are applied for the whole source file. It seems exactly the reason of your problem: your test, which calls the inline function, is compiled without coverage flags. So coverage counters aren't updated for that calls. When main.cpp, compiled with coverage flags, calls the inline function, this updates coverage counters.

So it seems that the test files I'm using DO need the coverage flags, but I can't apply them without a compilation freeze. As such, I'm at a loss for what to do. How can I speed up the compilation of my code?

Steps to reproduce

If you want an example of the compilation freeze with Catch, then follow the build steps in arnavb/test-codecov.

Extra information

Thanks for taking the time to read this; I know it's a lot!

arnavb commented 6 years ago

UPDATE:

Well, this problem no longer exists for me, since I switched to doctest which was much faster and could handle the flags.

I'm still leaving this issue open since it isn't resolved yet.

JoeyGrajciar commented 5 years ago

Can not reproduce with 2.5.0.