search

mull-project / mull

Practical mutation testing and fault injection for C and C++
https://mull.readthedocs.io
Apache License 2.0
734 stars 75 forks source link

Mull cannot run CTest's tests #794

Open ligurio opened 3 years ago

ligurio commented 3 years ago

CMake allows to generate binaries with unit tests from a C functions and these tests supports special protocol to run testcases [1]. To run a single testcase one need to execute a binary and choose a number of testcase:

(venv) sergeyb@pony:~/sources/FreeRDP/build$ ./Testing/TestPipe 
Available tests:
  0. TestPipeCreatePipe
  1. TestPipeCreateNamedPipe
  2. TestPipeCreateNamedPipeOverlapped
To run a test, enter the test number: 1
Server ReadFile: 32 bytes
Client ReadFile: 32 bytes
(venv) sergeyb@pony:~/sources/FreeRDP/build$

or run specify arbitrary name of testcase to run:

(venv) sergeyb@pony:~/sources/FreeRDP/build$ ./Testing/TestPipe -R TestPipeCreateNamedPipe
Server ReadFile: 32 bytes
Client ReadFile: 32 bytes
(venv) sergeyb@pony:~/sources/FreeRDP/build$

Also there was a MR to run all testcases at once [2], but it was not merged to a master though.

FreeRDP project actively uses tests written to run with CTest. Test failed with timeout when running using Mull because Mull don't know that test expects an input with a number of testcase.

$ git clone https://github.com/FreeRDP/FreeRDP/
$ cmake -DCMAKE_C_FLAGS="-fembed-bitcode -g -O0" -DCMAKE_CXX_FLAGS="-fembed-bitcode -g -O0" -DCMAKE_BUILD_TYPE=Debug -DCMAKE_EXPORT_COMPILE_COMMANDS=ON  -DCMAKE_C_COMPILER="/usr/bin/clang" -DCMAKE_CXX_COMPILER="/usr/bin/clang++" -DBUILD_TESTING=ON ..
$ make -j
$ mull-cxx -test-framework=CustomTest --ld-search-path=/lib/x86_64-linux-gnu --compdb-path=compile_commands.json  --reporters=Elements ./Testing/TestAsn1  
[info] Extracting bitcode from executable (threads: 1)
       [################################] 1/1. Finished in 2ms
[info] Loading bitcode files (threads: 8)
       [################################] 13/13. Finished in 11ms
[info] Compiling instrumented code (threads: 8)
       [################################] 13/13. Finished in 10ms
[info] Loading dynamic libraries (threads: 1)
       [################################] 1/1. Finished in 1ms
[info] Searching tests (threads: 1)
       [################################] 1/1. Finished in 0ms
[info] Preparing original test run (threads: 1)
       [################################] 1/1. Finished in 2ms
[info] Running original tests (threads: 1)
[warning] Original test failed
test: main
status: Timedout
stdout: 'Available tests:
  0. TestAsn1Module
  1. TestAsn1Encoder
  2. TestAsn1Decoder
  3. TestAsn1Encode
  4. TestAsn1Decode
  5. TestAsn1String
  6. TestAsn1Integer
  7. TestAsn1Compare
  8. TestAsn1BerEnc
  9. TestAsn1BerDec
 10. TestAsn1DerEnc
 11. TestAsn1DerDec
To run a test, enter the test number: '
stderr: ''

       [################################] 1/1. Finished in 3014ms
[info] No mutants found. Mutation score: infinitely high
[info] Total execution time: 3077ms
  1. https://gitlab.kitware.com/cmake/community/-/wikis/doc/ctest/Testing-With-CTest#running-individual-tests
  2. https://gitlab.kitware.com/cmake/cmake/-/merge_requests/3661#note_617780
ligurio commented 3 years ago

Also there was a MR to run all testcases at once [2], but it was not merged to a master though.

In upcoming cmake release it will be possible to run all testcases in a single test run https://github.com/Kitware/CMake/commit/3f6ff4b5dba01f2f58dca75546ebefd0830110a5

(I think it will be helpful if Mull's authors will decide to support tests generated by CMake.)