Test labels support

[Risto97]

Hello,

I would like to be able to label a group of tests so I could run them with ctest -L add_ip_fail

I attempted something like this, but it doesn't work:

set(TEST_NAME add_ip_fail_3_tokens_vendor_lib_ip)
ct_add_test(NAME ${TEST_NAME} EXPECTFAIL)
function(${${TEST_NAME}})
    add_ip(vendor::lib::ip)
endfunction()
set_property(TEST ${TEST_NAME} PROPERTY LABELS add_ip_fail)

Any solutions to this?

[ryanmrichard]

@Risto97 labels sound like a great idea!

It's admittedly been a bit since I looked at how ct_add_test worked, but I don't think ct_add_test registers the test with CTest, which is why CTest doesn't see it. IIRC, right now the test that is registered with CTest is something like cmake file_to_test.cmake where file_to_test.cmake is the file containing the code like your example and that happens in ct_add_dir.

To implement what you want we'd either have to settle for labels to apply to files (which would be fairly straightforward to implement) or we'd have to change how we register tests with CTest and have each ct_add_test command be a separate CTest test. I suspect the latter solution would be better in the long run, though I'm not sure how much work it would actually be.

IIRC @zachcran just updated how the tests are named, so maybe he can help you with a temporary solution along the lines of:

ct_add_dir(dir_to_test)
set_property(TEST ${name_cmaketest_assigned_to_file} PROPERTY LABELS add_ip_fail)

[Risto97]

Hello @ryanmrichard .

Thank you for the tip. I think labelling the files is sufficient for my needs and for most needs perhaps?

What you propose here:

ct_add_dir(dir_to_test)
set_property(TEST ${name_cmaketest_assigned_to_file} PROPERTY LABELS add_ip_fail)

Doesn't work for me, as I have multiple files in the directory where I call ct_add_dir()

$ tree ../tests/add_ip/
../tests/add_ip/
├── add_ip_1_token.cmake
├── add_ip.cmake
├── add_ip_description.cmake
├── add_ip_fail_2_tokens.cmake
├── add_ip_fail_3_tokens.cmake
├── add_ip_fail_4_tokens.cmake
└── add_ip_fail_too_many_tokens.cmake

1 directory, 7 files

I am playing with a solution with an additional LABEL argument to the ct_add_dir() function (https://github.com/Risto97/CMakeTest/commit/045f44b99e9801d913d28009b9c9ccd4e99ed11d)

The function call looks like this

ct_add_dir(add_ip USE_REL_PATH_NAMES 
    LABEL add_ip)

However as you see most likely you will be using the same LABEL as a directory name, so maybe another option is to have an argument like DIR_NAME_LABEL, that will use the first argument as label, or maybe just make a label by default?

ct_add_dir(add_ip USE_REL_PATH_NAMES DIR_NAME_LABEL)

Having this gives a nice report at the end:

Test project /tools/SoCMake/build
      Start  1: SoCMake_testing::add_ip/add_ip.cmake
 1/25 Test  #1: SoCMake_testing::add_ip/add_ip.cmake .................................................   Passed    0.15 sec
      Start  2: SoCMake_testing::add_ip/add_ip_1_token.cmake
 2/25 Test  #2: SoCMake_testing::add_ip/add_ip_1_token.cmake .........................................   Passed    0.15 sec
      Start  3: SoCMake_testing::add_ip/add_ip_description.cmake
 3/25 Test  #3: SoCMake_testing::add_ip/add_ip_description.cmake .....................................   Passed    0.15 sec
      Start  4: SoCMake_testing::add_ip/add_ip_fail_2_tokens.cmake
 4/25 Test  #4: SoCMake_testing::add_ip/add_ip_fail_2_tokens.cmake ...................................   Passed    2.09 sec
      Start  5: SoCMake_testing::add_ip/add_ip_fail_3_tokens.cmake
 5/25 Test  #5: SoCMake_testing::add_ip/add_ip_fail_3_tokens.cmake ...................................   Passed    2.36 sec
      Start  6: SoCMake_testing::add_ip/add_ip_fail_4_tokens.cmake
 6/25 Test  #6: SoCMake_testing::add_ip/add_ip_fail_4_tokens.cmake ...................................   Passed    1.49 sec
      Start  7: SoCMake_testing::add_ip/add_ip_fail_too_many_tokens.cmake
 7/25 Test  #7: SoCMake_testing::add_ip/add_ip_fail_too_many_tokens.cmake ............................   Passed    0.92 sec
      Start  8: SoCMake_testing::ip_include_directories/ip_include_directories.cmake
 8/25 Test  #8: SoCMake_testing::ip_include_directories/ip_include_directories.cmake .................   Passed    0.10 sec
      Start  9: SoCMake_testing::ip_include_directories/ip_include_directories_get_multilang.cmake
 9/25 Test  #9: SoCMake_testing::ip_include_directories/ip_include_directories_get_multilang.cmake ...   Passed    0.10 sec
      Start 10: SoCMake_testing::ip_compile_definitions/ip_compile_definitions.cmake
10/25 Test #10: SoCMake_testing::ip_compile_definitions/ip_compile_definitions.cmake .................   Passed    0.10 sec
      Start 11: SoCMake_testing::ip_compile_definitions/ip_compile_definitions_get_multilang.cmake
11/25 Test #11: SoCMake_testing::ip_compile_definitions/ip_compile_definitions_get_multilang.cmake ...   Passed    0.10 sec
      Start 12: SoCMake_testing::ip_compile_definitions/ip_compile_definitions_hier.cmake
12/25 Test #12: SoCMake_testing::ip_compile_definitions/ip_compile_definitions_hier.cmake ............   Passed    0.10 sec
      Start 13: SoCMake_testing::alias_dereference/alias_dereference.cmake
13/25 Test #13: SoCMake_testing::alias_dereference/alias_dereference.cmake ...........................   Passed    0.10 sec
      Start 14: SoCMake_testing::check_languages/check_languages_additional.cmake
14/25 Test #14: SoCMake_testing::check_languages/check_languages_additional.cmake ....................   Passed    0.10 sec
      Start 15: SoCMake_testing::check_languages/check_languages_fatal.cmake
15/25 Test #15: SoCMake_testing::check_languages/check_languages_fatal.cmake .........................   Passed    0.34 sec
      Start 16: SoCMake_testing::check_languages/check_languages_standard.cmake
16/25 Test #16: SoCMake_testing::check_languages/check_languages_standard.cmake ......................   Passed    0.10 sec
      Start 17: SoCMake_testing::check_languages/check_languages_warning.cmake
17/25 Test #17: SoCMake_testing::check_languages/check_languages_warning.cmake .......................   Passed    0.10 sec
      Start 18: SoCMake_testing::ip_sources/ip_source_fail.cmake
18/25 Test #18: SoCMake_testing::ip_sources/ip_source_fail.cmake .....................................   Passed    0.34 sec
      Start 19: SoCMake_testing::ip_sources/ip_sources.cmake
19/25 Test #19: SoCMake_testing::ip_sources/ip_sources.cmake .........................................   Passed    0.10 sec
      Start 20: SoCMake_testing::ip_sources/ip_sources_get_multilang.cmake
20/25 Test #20: SoCMake_testing::ip_sources/ip_sources_get_multilang.cmake ...........................   Passed    0.10 sec
      Start 21: SoCMake_testing::ip_link/ip_link1.cmake
21/25 Test #21: SoCMake_testing::ip_link/ip_link1.cmake ..............................................   Passed    0.10 sec
      Start 22: SoCMake_testing::ip_link/ip_link_self_link_fatal.cmake
22/25 Test #22: SoCMake_testing::ip_link/ip_link_self_link_fatal.cmake ...............................   Passed    0.34 sec
      Start 23: iverilog_simple_test
23/25 Test #23: iverilog_simple_test .................................................................   Passed    0.00 sec
      Start 24: iverilog_multi_top
24/25 Test #24: iverilog_multi_top ...................................................................   Passed    0.00 sec
      Start 25: peakrdl_print_test
25/25 Test #25: peakrdl_print_test ...................................................................   Passed    0.41 sec

100% tests passed, 0 tests failed out of 25

Label Time Summary:
add_ip                    =   7.31 sec*proc (7 tests)
alias_dereference         =   0.10 sec*proc (1 test)
check_languages           =   0.62 sec*proc (4 tests)
ip_compile_definitions    =   0.29 sec*proc (3 tests)
ip_include_directories    =   0.20 sec*proc (2 tests)
ip_link                   =   0.44 sec*proc (2 tests)
ip_sources                =   0.53 sec*proc (3 tests)
peakrdl                   =   0.41 sec*proc (1 test)

Total Test time (real) =   9.92 sec

[ryanmrichard]

@Risto97 I really like your ct_add_dir(add_ip USE_REL_PATH_NAMES DIR_NAME_LABEL) solution!

I agree with you that using the directory name is probably what most people want, so I'm also fine just using LABEL instead of DIR_NAME_LABEL as the flag/toggle. If down the road someone wanted to specify a different label name we could add an option like LABEL_NAME which takes one argument.

[Risto97]

Sounds good.

Check if implementation #113 is what you meant.

[ryanmrichard]

@Risto97 Thank you so much for the contribution!