search

tcunit / TcUnit-Runner

Program that makes it possible to automate runs of TcUnit unit tests
Other
34 stars 17 forks source link

Building the project in TCXaeShell does not collect test results correctly when generating the XML file. #21

Closed Traavaas closed 3 years ago

Traavaas commented 3 years ago

TwinCAT3 Version: 3.1.4024.10 Visual Studio Development Tools Environment (DTE) version: TcXaeShell.DTE.15.0 TcUnit Version: 1.1.0.0 TcUnit-Runner Version: 0.9.3.0

When building the project in Visual Studio 2013, the XML file looks right. This is what it looks like: 

<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4">
  <testsuite id="0" name="PRG_MAIN.fb_Subtract_Test" tests="2" failures="0">
    <testcase name="FiveMinusThreeEqualsTwo" assertions="1" classname="PRG_MAIN.fb_Subtract_Test" status="PASS" />
    <testcase name="TwelveMinusFiveEqualsSeven" assertions="1" classname="PRG_MAIN.fb_Subtract_Test" status="PASS" />
  </testsuite>
  <testsuite id="1" name="PRG_MAIN.fb_Sum" tests="2" failures="0">
    <testcase name="TwoPlusTwoEqualsFour" assertions="1" classname="PRG_MAIN.fb_Sum" status="PASS" />
    <testcase name="ZeroPlusZeroEqualsZero" assertions="1" classname="PRG_MAIN.fb_Sum" status="PASS" />
  </testsuite>
</testsuites>

However, when it tries building in TCXaeShell, the XML file looks like this: 

<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4" />

When it tries to record that file in Jenkins, My pipeline outputs this: 

[Pipeline] junit
Recording test results
None of the test reports contained any result

When I build and run the project in TCXaeShell itself (away from Jenkins and TcUnitRunner), I am able to see the test cases and details in the output and Error List.

sagatowski commented 3 years ago

Thanks for your report @Traavaas.

Is it possible for you to upload the project here? I'll try to reproduce on my end.

Traavaas commented 3 years ago

@sagatowski Appreciate your help.

sagatowski commented 3 years ago

@Traavaas Which version of TcUnit are your libraries/TcProjects using? 1.1 or the current trunk?

sagatowski commented 3 years ago

Also, which version of TcUnit-Runner?

Traavaas commented 3 years ago

Updated original post with those details!

sagatowski commented 3 years ago

Hey @Traavaas!

I have now done the following: Created a fresh new machine, with:

I noticed there were two tasks in the example project, so TcUnit-Runner needs to know which one to run (if there is only one it will run that one), so this needs to be provided as input argument (it's described in the documentation if you want to know all details). Note the -t PlcTask argument provided

I executed TcUnit-Runner in this way and got the following result:

C:\Program` Files (x86)\TcUnit-Runner>TcUnit-Runner.exe -v "C:\Code\SampleProject\SimpleExampleProject\SimpleTcUnitExampleProject.sln" -t PlcTask
2021-03-24 19:27:56 - TcUnit-Runner build: 0.9.3.0
2021-03-24 19:27:56 - TcUnit-Runner build date: 2020-12-22
2021-03-24 19:27:56 - Visual Studio solution path: C:\Code\SampleProject\SimpleExampleProject\SimpleTcUnitExampleProject.sln
2021-03-24 19:27:56 -
2021-03-24 19:27:56 - In TwinCAT project file, found TwinCAT version 3.1.4024.10
2021-03-24 19:27:56 - In Visual Studio solution file, found visual studio version 15.0
2021-03-24 19:27:56 - Version is pinned: False
2021-03-24 19:27:56 - Trying to load the Visual Studio Development Tools Environment (DTE) version 'TcXaeShell.DTE.15.0' ...
2021-03-24 19:28:03 - ...SUCCESSFUL!
2021-03-24 19:28:15 - Using the TwinCAT remote manager to load TwinCAT version '3.1.4024.10'...
2021-03-24 19:28:33 - Setting task 'PlcTask' enable and autostart, and all other tasks (if existing) to disable and non-autostart
2021-03-24 19:28:57 - Setting target NetId to '127.0.0.1.1.1'
2021-03-24 19:28:57 - Enabling boot project and setting BootProjectAutostart on 127.0.0.1.1.1
2021-03-24 19:29:43 - Waiting for results from TcUnit...
2021-03-24 19:29:59 - ... got 32 report lines so far.
2021-03-24 19:29:59 - All results from TcUnit obtained
2021-03-24 19:30:14 - Done collecting TC results
2021-03-24 19:30:14 - Writing xUnit XML file to C:\Code\SampleProject\SimpleExampleProject\TcUnit_xUnit_results.xml
2021-03-24 19:30:14 - Closing the Visual Studio Development Tools Environment (DTE)...
2021-03-24 19:30:44 - Exiting application...

And this is the final end-result (in the xml-file)

<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4">
  <testsuite id="0" name="PRG_TEST.fb_Subtract_Test" tests="2" failures="0">
    <testcase name="FiveMinusThreeEqualsTwo" assertions="1" classname="PRG_TEST.fb_Subtract_Test" status="PASS" />
    <testcase name="TwelveMinusFiveEqualsSeven" assertions="1" classname="PRG_TEST.fb_Subtract_Test" status="PASS" />
  </testsuite>
  <testsuite id="1" name="PRG_TEST.fb_Sum" tests="2" failures="0">
    <testcase name="TwoPlusTwoEqualsFour" assertions="1" classname="PRG_TEST.fb_Sum" status="PASS" />
    <testcase name="ZeroPlusZeroEqualsZero" assertions="1" classname="PRG_TEST.fb_Sum" status="PASS" />
  </testsuite>
</testsuites>

In other words, all worked perfectly fine.

Did you do anything different than above?

I've had a long day at work today so I will go to sleep now, but if you do any findings please write them down here and I'll try to answer as soon as I can.

/ J

Traavaas commented 3 years ago

I executed the program the same way you did.

C:\Program Files (x86)\TcUnit-Runner>TcUnit-Runner.exe -v "C:\Users\tyu\tcunitargumentstest\SimpleExampleProject\SimpleTcUnitExampleProject.sln" -t PlcTask
2021-03-24 19:49:44 - TcUnit-Runner build: 0.9.3.0
2021-03-24 19:49:44 - TcUnit-Runner build date: 12/22/2020
2021-03-24 19:49:44 - Visual Studio solution path: C:\Users\tyu\tcunitargumentstest\SimpleExampleProject\SimpleTcUnitExampleProject.sln
2021-03-24 19:49:44 -
2021-03-24 19:49:44 - In TwinCAT project file, found TwinCAT version 3.1.4024.10
2021-03-24 19:49:44 - In Visual Studio solution file, found visual studio version 15.0
2021-03-24 19:49:44 - Version is pinned: False
2021-03-24 19:49:44 - Trying to load the Visual Studio Development Tools Environment (DTE) version 'TcXaeShell.DTE.15.0' ...
2021-03-24 19:49:46 - ...SUCCESSFUL!
2021-03-24 19:49:49 - Using the TwinCAT remote manager to load TwinCAT version '3.1.4024.10'...
2021-03-24 19:49:56 - Setting task 'PlcTask' enable and autostart, and all other tasks (if existing) to disable and non-autostart
2021-03-24 19:50:06 - Setting target NetId to '127.0.0.1.1.1'
2021-03-24 19:50:06 - Enabling boot project and setting BootProjectAutostart on 127.0.0.1.1.1
2021-03-24 19:50:35 - Waiting for results from TcUnit...
2021-03-24 19:50:45 - ... got 6 report lines so far.
2021-03-24 19:50:45 - All results from TcUnit obtained
2021-03-24 19:50:55 - Done collecting TC results
2021-03-24 19:50:55 - Writing xUnit XML file to C:\Users\tyu\tcunitargumentstest\SimpleExampleProject\TcUnit_xUnit_results.xml
2021-03-24 19:50:55 - Closing the Visual Studio Development Tools Environment (DTE)...
2021-03-24 19:51:15 - Exiting application...

My xml file looks like this: 

<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4" />

Any suggestions?

sagatowski commented 3 years ago

@Traavaas That is really strange. Strange indeed. It's as if it only sees the final test results (how many tests have been running and failed, succeeded etc), but not the individual test results. I have managed to re-produce your error, but then I had to manually go into the project and change this parameter To FALSE:

But it has to be TRUE, which is the default behavior, so I guess you have not changed this to FALSE? image

If I set the above to FALSE, then I get this:

<?xml version="1.0" encoding="utf-8"?>
<testsuites failures="0" tests="4" />

Also, do you have any other versions of TwinCAT installed? You installed the version 1.1 from RELEASES?

Just trying to understand what differs between our machines now...

Traavaas commented 3 years ago

Yeah, I have not messed with TcUnit itself. Mine looks exactly like yours.

image

I do not have any other versions of TwinCAT installed. I definitely installed the version from RELEASES. I compared the sha-1 of the compiled-library of the one I have on my computer to the one I just download now, and they happen to be the same.

sagatowski commented 3 years ago

@Traavaas Also, if you run the project manually in TcXaeShell, do you get the following messages marked in red?

image

The messages are the ones that are used to gather information on test results

Traavaas commented 3 years ago

I do get the messages marked in red. image

sagatowski commented 3 years ago

Everything looks in order... so TcUnit-Runner should get all results and all should be fine. I will have to sleep on this one and think about it.

I might add some verbose debug logs to the TcUnit-Runner and do a special-compile for you (or I might simply add a -verbose mode to the logger), and then you could return the output to me?

Traavaas commented 3 years ago

Yeah that sounds good. Thanks again for your help!

sagatowski commented 3 years ago

@Traavaas Enabling debug-level in log4net on the fly was a little trickier than I thought. It will have to be a weekend-project. I'll keep you updated.

sagatowski commented 3 years ago

Alright, I have implemented a debug level of logging. Actually re-did many parts of the logging while I was at it. I'm much more happy with the logging now 😄 

Options:
  -v, --VisualStudioSolutionFilePath=VALUE
                             The full path to the TwinCAT project (sln-file)
  -t, --TcUnitTaskName=VALUE [OPTIONAL] The name of the task running TcUnit
                               defined under "Tasks"
  -a, --AmsNetId=VALUE       [OPTIONAL] The AMS NetId of the device of where
                               the project and TcUnit should run
  -w, --TcVersion=VALUE      [OPTIONAL] The TwinCAT version to be used to
                               load the TwinCAT project
  -u, --Timeout=VALUE        [OPTIONAL] Timeout the process with an error
                               after X minutes
  -d, --debug                [OPTIONAL] Increase debug message verbosity
  -?, -h, --help

So simply add -d when executing. I haven't made a release of this yet, so simply build TcUnit-Runner from source. Run the project again (with -d), and upload the output file (in /logs) from that run here. Attached is my output from the run of your TwinCAT solution. 2021-03-27_09_27_24.log

See commit 324fd9f & e6bac31.

Traavaas commented 3 years ago

Can you explain how to build TcUnit-Runner from source? I've cloned the repository to my computer so far.

sagatowski commented 3 years ago

Can you explain how to build TcUnit-Runner from source? I've cloned the repository to my computer so far.

It's described on the main page: https://github.com/tcunit/TcUnit-Runner Go to "Want to contribute?"

Traavaas commented 3 years ago

@sagatowski Thanks for taking the time to help me! I was able to to build and run the project with the -d argument perfectly. I'm not sure what exactly changed but after building and running from source I ran my project with the release version and it happened to work perfectly as well. It was also able to record the results in Jenkins. I was also able to verify the the results are correct in the xml file. This is very odd because I did not make a single change to the project over the weekend. I believe this may be a problem of my computer rather than TcUnitRunner.

2021-03-29_18_42_23.log

sagatowski commented 3 years ago

That's weird! At what device were you running the TcUnit-Runner? VirtualBox? Your PC? PLC? Do you have any idea what the issue with the computer might have been?

Traavaas commented 3 years ago

I'm using my PC. I've been trying to figure out the reason why it randomly started working and I have no clue. I re-downloaded the zip file I uploaded to you and ran tcunit-runner (The release version that I have always been using without the new debug feature) on it. It ended up working correctly! It makes no sense to me haha

sagatowski commented 3 years ago

Alright, one time is no time. I'll close this issue. If you encounter this problem again, upload a debug log here and we'll re-open it.