search

bneumann / CppUTest-Test-Adapter

A test adapter for Visual Studio Code for the C/C++ Unit Test framework CppUTest.
MIT License
16 stars 7 forks source link

Regex parser for CppUTest fails when debug stuff printed out by the test application #11

Closed MedadRufus closed 2 years ago

MedadRufus commented 3 years ago

First of all, many thanks for developing this convenient tool to work with vs code. I am currently using the latest version of CppUTest v4.0 and CppUTest-Test-Adapter 0.1.12.

The problem arises when some debug text is printed out while running the test. For example, the CppUTest adapter incorrectly parses the following output and marks as failing:

medad@medad-ThinkPad-P51:~/Downloads/cpputest_example/build/tests/pythagorean_ut$ /home/medad/Downloads/cpputest_example/build/tests/pythagorean_ut/pythagorean_ut -v
TEST(Pythagorean, BlackBoxTest)
SquareRoot_sqrt gets called with parameter: 100.000000. <-- debug printout from test app
 - 0 ms
TEST(Pythagorean, simpleTest)
SquareRoot_sqrt gets called with parameter: 25.000000.  <-- debug printout from test app
 - 0 ms

OK (2 tests, 2 ran, 6 checks, 0 ignored, 0 filtered out, 0 ms)

Conversely, the following output is correctly parsed, with no extra text inbetween:

medad@medad-ThinkPad-P51:~/Downloads/cpputest_example/build/tests/pythagorean_ut$ /home/medad/Downloads/cpputest_example/build/tests/pythagorean_ut/pythagorean_ut -v
TEST(Pythagorean, BlackBoxTest) - 0 ms
TEST(Pythagorean, simpleTest) - 0 ms

OK (2 tests, 2 ran, 6 checks, 0 ignored, 0 filtered out, 0 ms)

I suggest 3 solutions:

  1. Optimal: Switch back to using output to JUnit style xml files. I confirm that when I revert CppUTest adapter back to 0.1.6, I no longer have the parsing issue. Only after that(80e99550a5c75586263b44ff274d80838f35512a) was the switch done from outputting JUnit style xml files to simple verbose output. Using the JUnit style will be better because it is a standardised solution, and a parser does not have to be written/maintained.
  2. Suboptimal solution: Update the regex parser to handle the messy stdout log.
  3. Suboptimal workaround: simply do not print out anything from the test application. So the regex parser does not get confused.
MedadRufus commented 3 years ago

I will be happy to work on the code and solution implementation

bneumann commented 3 years ago

Wow, thanks for the detailed issue. Yeah I figured the day may come where some stdout will confuse my regex parser 😉 I am having some days off so every help is welcome. If you find a way to not clutter the output directory with the xmls it would be a perfect solution. But if not it still is the best approach. You could base of your work from the latest branch feat/addingLogger

MedadRufus commented 3 years ago

Thanks, I will experiment with a solution this week.

MedadRufus commented 3 years ago

Just discussing the solution. What if we treat the junit xml output as a temp file, and delete it after the test adaptor has all the info. Will that be a good practice? That will ensure no file clutter.

I have yet to see what the output looks like though.

bneumann commented 3 years ago

Yeah was thinking about that too. I was afraid that people are using these files and we would delete them. But yeah would be the way to go