codereport / jsource

J Language Source Code. Livestream links ⬇️
https://www.youtube.com/playlist?list=PLVFrD1dmDdvfVhYLU_iKkV67X9XqCJLWe
Other
38 stars 20 forks source link

Group unit tests into slow and fast tests #98

Closed Sebanisu closed 3 years ago

Sebanisu commented 3 years ago

To filter/sort tests by time the lists clion has a dump test results to xml. Then I used a site that converts xml to csv. Once I got all the data grouped up I copied the columns into the CMakeLists.txt.


Currently have the three lists. One for fast test, one for slow tests and I put the disabled tests into their own list. I created a function out of the for each loop. So you can choose what list gets added and what the prefix on the testname is.

function(add_tests test_list prefix)
foreach (test_name IN LISTS ${test_list})
    file(GENERATE
            OUTPUT "$<${is_multi_config}:$<CONFIG>/>${test_name}.ijs"
            CONTENT "0!:0 <'${CMAKE_CURRENT_SOURCE_DIR}/tsu.ijs'
LIBTSDLL=:'$<TARGET_FILE:tsdll> '
exit -.&(0!:3) <testpath,'${test_name}.ijs'"
            CONDITION $<BOOL:${BUILD_TESTING}>
            )
    add_test(
            NAME ${prefix}${test_name}
            COMMAND jconsole "$<${is_multi_config}:$<CONFIG>/>${test_name}.ijs"
    )
endforeach ()
endfunction()
add_tests(fast_test_cases fast_)
add_tests(slow_test_cases slow_)
add_tests(disabled_test_cases disabled_)

Running tests works fine:

ninja -C build test # works as normal

To only run fast or slow tests:

cd ./build && /usr/bin/ctest --force-new-ctest-process -C Debug -R hare_ && cd .. #fast
cd ./build && /usr/bin/ctest --force-new-ctest-process -C Debug -R tortoise_ && cd .. #slow

Disabled Tests Note

The disabled_test_cases are still commented out but if you wanted to uncomment them you could run them only by using

cd ./build && /usr/bin/ctest --force-new-ctest-process -C Debug -R disabled_ && cd ..

Removed Add Custom Target

I had a add_custom_target but it wasn't showing output when ran from ninja and clion didn't know what to do with it. So I removed it from the CMakeLists.txt. If someone knows a way to make it work, than you don't need to run ctest directly.

add_custom_target(fast_tests COMMAND ${CMAKE_CTEST_COMMAND} -R hare_ -C Debug) #fast
add_custom_target(slow_tests COMMAND ${CMAKE_CTEST_COMMAND} -R tortoise_ -C Debug) #slow
ninja -C build fast_tests # shows no output
ninja -C build slow_tests # shows no output

resolves https://github.com/codereport/jsource/issues/16 referenced prefix idea from https://stackoverflow.com/a/28710462/2588183

Sebanisu commented 3 years ago

No prob I'll fix it in a bit. Taking a break.

Sebanisu commented 3 years ago
        gmmf      # 0.150 seconds, Exception: SegFault, Only on Windows, #bus errors, unstable on CI?
        gmmf1s    # 0.110 seconds, Exception: SegFault, Only on Windows
        gmmf1u    # 0.260 seconds, Exception: SegFault, Only on Windows
        gmmf1w    # 0.110 seconds, Exception: SegFault, Only on Windows

I enabled all tests and ran on the CI these passed so i removed them from disabled. Though maybe they were disabled because they were unstable tests per comment? Anything that passed on CI when i tried I had enabled. So things might need to be disabled again. I meant to put this on my pr. Sorry I had posted this on the issue.

So this should be better now more tests were moved to slow around 3-4 seconds on my pc.

image