Enhancement: Adding Check test discovery for CMake

[phetdam]

Hi,

Been using Check standalone with Makefiles but have recently switched to using CMake to build my C projects.

I noticed there is no Check equivalent of Google Tests's gtest_discover_tests or gtest_add_tests. These are useful for making CMake aware of the individual tests that are registered with the test runner, as otherwise CMake considers a single test runner to be a single unit test. So no matter how many test cases and suites you have in the runner, ctest says only 1 test succeeds/fails.

It's possible to run by suite with CK_RUN_SUITE, by case with CK_RUN_CASE or by using tags and CK_(INCLUDE|EXCLUDE)_TAGS, but these are environment variables so in the CMakeLists.txt, I would still have to do something like

# let's say there are two test cases with a few tests each, case_1 and case_2, in one suite
add_executable(test_runner [source files])
# i chose to link dynamically against Check, i.e. checkDynamic.dll on Windows,
# libcheck.so on *nix. dug through the check-targets.cmake and realized i could do
# something like find_package(Check PATHS $ENV{CHECK_ROOT}), where CHECK_ROOT
# can be set from env to locate the install path, so i could use the aliased targets.
target_link_libraries(test_runner Check::checkShared [other libs])
# register case_1, case_2 as individual "tests" with CMake
add_test(
    NAME test_case_1
    COMMAND sh -c "CK_RUN_CASE=case_1 && $<TARGET_FILE:test_runner>"
)
add_test(
    NAME test_case_2
    COMMAND sh -c "CK_RUN_CASE=case_2 && $<TARGET_FILE:test_runner>"
)

Note that this would only work on a *nix-like platform since I used sh. I also can't just pass the line passed to sh directly since CMake then tries to look for that string as an executable name. The other option, which is wrapping this into a shell script, is of course still a manual approach. To get things to work on Windows I would have to do something like

if(WIN32)
    add_test(
        NAME test_case_1
        COMMAND cmd /c "set CK_RUN_CASE=case_1 && $<SHELL_PATH:$<TARGET_FILE:test_runner>> && set CK_RUN_CASE="
    )
    # more Windows stuff
else()
    # *nix stuff
endif()

I couldn't actually get this to work; cmd was complaining that the syntax was wrong (works fine when copied directly into the command line). Either way, both of these approaches are hacky af and Check test granularity is only down to the case level.

gtest_discover_tests seems difficult to replicate given the current design of Check, since this just reads the output from using the --gtest_list_tests argument since Google Test runners can accept some CLI arguments. gtest_add_tests on the other hand scans source files for tests using a regex match, which although in the Google Test case has some shortcomings, might work just fine for Check test cases. After all, all the tests have the form START_TEST(test_name) { /* test body */ } END_TEST.

Could this be an enhancement added to Check to better support its integration with CMake? Currently, besides the CMake fumbling, the only other hacky workaround I can think of would be to split tests into individual cases and suites with their own main by using a macro or something, and fixtures, setup functions, etc. would have to be shared through header files.

However, maybe I'm missing something that Check already supports for CMake OOTB. If that's the case, please let me know.