search

tcunit / TcUnit

An unit testing framework for Beckhoff's TwinCAT 3
Other
258 stars 72 forks source link

Integration to Jenkins/build automation server #7

Closed sagatowski closed 4 years ago

sagatowski commented 5 years ago

Today it's only possible to run the tests locally on a developer/engineering machine. In large projects this is usually not optimal, as the amount of source code can be overwhelmingly large.

In the sphere of continous integration and continous delivery there needs to be a more automated way to run the tests. One solution/suggestion is to write a program/script that automatically runs the tests on the build-server using a combination of the Microsoft Visual Studio development tools environment plus the TwinCAT automation interface.

teddyrendahl commented 5 years ago

Awesome repository! Came here looking for this so just putting a ⬆️. Would love to have an out of the box solution to run on Appveyor.

edgargrimberg commented 5 years ago

Great initiative. A bit offtopic, but I was wondering if there is a simple way to just invoke msbuild to build the tcproj on the buildserver, and encode multiple configurations to include / exclude the unit tests. Think a Jenkins builder that can be trigger a compilation to produce the build artifacts, and another compilation to create the binaries with unit tests, to be run later on.

klauer commented 4 years ago

@edgargrimberg I tried and failed to get msbuild to work, but have made some progress with devenv:

"%ProgramFiles(x86)%\Microsoft Visual Studio 12.0\Common7\IDE\devenv.exe" (your_twincat_solution).sln /Rebuild "Debug|TwinCAT RT (x86)" /Out build_output.txt

Devenv unfortunately runs in the background - returning immediately after you type the command - but eventually the build results will show up in build_output.txt.

u-hafner commented 4 years ago

@klauer Maybe this can help for blocking until it ends: https://stackoverflow.com/questions/10416122/for-command-using-devenv-exe-runs-too-quickly

kumaraswamygaviyappa commented 4 years ago

Thank you @sagatowski, I could able to build a job in Jenkins as you explained in the series: https://bit.ly/2Y0TBIJ

As the next task, i am trying to add unit testing using TcUnit to Jenkins. I could able to Activate Configuration of TwinCAT System. 

ITcSmTreeItem plcProjectRootItem = sysMan.LookupTreeItem("TIPC^UnitTestJenkins");
ITcPlcProject iecProjectRoot = (ITcPlcProject)plcProjectRootItem;
iecProjectRoot.BootProjectAutostart = true;
iecProjectRoot.GenerateBootProject(true);

sysMan.ActivateConfiguration();
sysMan.StartRestartTwinCAT();

Results of the unit testing appears on the error list window of VS2013 project(VS2013 was invoked by the C#):

Message 1   The application is up to date       0   0   
Message 2   Build complete -- 0 errors, 0 warnings : ready for download!        0   0   
Message 3   Generate TMC information ...        0   0   
Message 4   Import symbol information ...       0   0   
Message 5   generate boot information...    PLC.UnitTestJenkins 0   0   UnitTestJenkins (UnitTestJenkins\UnitTestJenkins)
Message 6   27.11.2019 11:17:27 875 ms | 'TwinCAT System' (10000): Ein Neustart des TwinCAT Systems wurde von AmsNetId: 10.73.8.239.1.1 Port 35688 angefordert.     
Message 7   27.11.2019 11:17:28 511 ms | 'TwinCAT System' (10000): Konfiguration des COM Servers TcEventLogger wird gesichert!      
Message 8   27.11.2019 11:17:29 465 ms | 'TwinCAT System' (10000): COM Servers TcEventLogger wird heruntergefahren!     
Message 9   27.11.2019 11:17:31 515 ms | 'TwinCAT System' (10000): Konfiguration des COM Servers TcEventLogger wird geladen!        
Message 10  27.11.2019 11:17:31 516 ms | 'TwinCAT System' (10000): COM Server TcEventLogger wird initialisiert!     
Message 11  27.11.2019 11:17:31 561 ms | 'TwinCAT System' (10000): TcIo Server wurde gestartet: TcIo.       
Message 12  27.11.2019 11:17:31 573 ms | 'TwinCAT System' (10000): TcPlc30 Server wurde gestartet: TcPlc30.     
Message 13  27.11.2019 11:17:31 580 ms | 'TwinCAT System' (10000): TcRtsObjects Server wurde gestartet: TcRtsObjects.       
Message 14  27.11.2019 11:17:31 586 ms | 'TwinCAT System' (10000): TcRTime Server wurde gestartet: TcRTime.     
Message 15  27.11.2019 11:17:31 705 ms | 'License Server' (30): license validation status is Valid(3)       
Message 16  27.11.2019 11:17:31 749 ms | 'TCRTIME' (200): Intel(R) Core(TM)-i 4'th generation detected      
Message 17  27.11.2019 11:17:32 160 ms | 'TwinCAT System' (10000): COM Server TcEventLogger wird gestartet!     
Error   18  27.11.2019 11:17:32 183 ms | 'PlcTask' (350): FAILED TEST 'PRG_TEST.fbSum_Test@TwoPlusTwoEqualsFour', EXP: 4, ACT: 3, MSG: The calculation is not correct       
Error   19  27.11.2019 11:17:32 243 ms | 'PlcTask' (350): FAILED TEST 'PRG_TEST.fbDegreesToRadians@DegreesToRadians_Test1', EXP: 0.0, ACT: 0.01745329, MSG: The calculation is not correct      
Error   20  27.11.2019 11:17:32 303 ms | 'PlcTask' (350): FAILED TEST 'PRG_TEST.fbDegreesToRadians@DegreesToRadians_Test2', EXP: 1.0, ACT: 0.01745329, MSG: The calculation is not correct      
Error   21  27.11.2019 11:17:32 463 ms | 'PlcTask' (350): | ==========TESTS FINISHED RUNNING==========      
Error   22  27.11.2019 11:17:32 783 ms | 'PlcTask' (350): | Test suites: 2      
Error   23  27.11.2019 11:17:33 103 ms | 'PlcTask' (350): | Tests: 4        
Error   24  27.11.2019 11:17:33 423 ms | 'PlcTask' (350): | Successful tests: 1     
Error   25  27.11.2019 11:17:33 743 ms | 'PlcTask' (350): | Failed tests: 3     
Error   26  27.11.2019 11:17:34 063 ms | 'PlcTask' (350): | ======================================

Using ErrorItems errors = dte.ToolWindows.ErrorList.ErrorItems; i could display only message 1 to 5. Other Messages has different format 'TwinCAT System' (10000) or 'PlcTask' (350):. Is this will not be captured by Edte.ToolWindows.ErrorList.ErrorItems;?

I tried other approach to see if thing will work out. In the Beckhoff website, i foundITcSysManager2::GetLastErrorMessages https://bit.ly/37DKmm9 method returns the last error messages.

I wore the code as below, ErrorMessage.Length returns always 0

ITcSysManager2 ErrorMessages = pro.Object;
string ErrorMessage = ErrorMessages.GetLastErrorMessages();
int test = ErrorMessage.Length;
Console.WriteLine("ErrorMessages.GetLastErrorMessages: " + test);

Is there any way to capture these messages on display on to the console.

klauer commented 4 years ago

@u-hafner - thanks! I can confirm that both of the following work, though devenv.com nicely tees the output to both standard error and the log file specified:

With devenv.com: "%ProgramFiles(x86)%\Microsoft Visual Studio 12.0\Common7\IDE\devenv.com" project.sln /Rebuild "Debug|TwinCAT RT (x86)" /Out logfile.log

Using start /wait: start /wait "" "%ProgramFiles(x86)%/Microsoft Visual Studio 12.0/Common7/IDE/devenv.exe" project.sln /Rebuild "Debug|TwinCAT RT (x86)" /Out logfile.log

kumaraswamygaviyappa commented 4 years ago

Thank you @sagatowski, I could able to build a job in Jenkins as you explained in the series: https://bit.ly/2Y0TBIJ

As the next task, i am trying to add unit testing using TcUnit to Jenkins. I could able to Activate Configuration of TwinCAT System.


ITcSmTreeItem plcProjectRootItem = sysMan.LookupTreeItem("TIPC^UnitTestJenkins");

Thank you @sagatowski Your suggestion helped to solve this issue. https://stackoverflow.com/questions/59101242/twincat3-automation-interface-display-o-p-on-jenkins-console/59161528#59161528

Aliazzzz commented 4 years ago

See also Export results in Xunit XML format #11

sagatowski commented 4 years ago

Parts of this functionality is now merged into the main by the PR #91. https://github.com/tcunit/TcUnit/pull/91

Aliazzzz commented 4 years ago

I think this issue is now solved

sagatowski commented 4 years ago

Not yet. For TcUnit, the documented way will be handled by the TcUnit-Runner, which is not yet released. The software is done, but the documentation is not yet finished - and don't want to release one without the other :-)

Aliazzzz commented 4 years ago

Looking forward to see your C# TcUnit-Runner in action! It must be great to finally run true CI/CD scenario's using it (allthough from CODESYS point of view it has little added value as that has (iron) python-power under the hood) Never the less a breakthrough for TwinCAT related software!

sagatowski commented 4 years ago

I'm looking forward to it as well. It's just been so much more challenging than I could ever have imagined. So many use cases to consider. Right now it's at least in such a state that it's ready to receive feedback. I can imagine much of this is already in place for vanilla codesys.

Aliazzzz commented 4 years ago

The feedback will help it get mature more quick, so that is a very good thing. Also, borrow ideas from the Python CODESYS API and the CODESYS.exe arguments. If you at least read up on them, you get an idea of what neat stuff you could add and fill in those gaps. I’ll follow the TcUnit-Runner with great interest.

sagatowski commented 4 years ago

Do you have a link to the documentation of this API?

Aliazzzz commented 4 years ago

Good starting point; https://help.codesys.com/webapp/_cds_struct_using_scripts;product=codesys;version=3.5.15.0

CODESYS Python scripting API; [https://help.codesys.com/webapp/idx-scriptingengine;product=ScriptEngine;version=3.5.15.0] (https://help.codesys.com/webapp/idx-scriptingengine;product=ScriptEngine;version=3.5.15.0)

CODESYS Commandline Interface, maybe TwinCAT has support for this too? [https://help.codesys.com/webapp/_cds_commandline;product=codesys;version=3.5.15.0] (https://help.codesys.com/webapp/_cds_commandline;product=codesys;version=3.5.15.0)

Roald87 commented 4 years ago

A bit offtopic, but I was wondering if there is a simple way to just invoke msbuild to build the tcproj on the buildserver, and encode multiple configurations to include / exclude the unit tests.

@edgargrimberg Its not possible with MsBuild, because plc projects are build with CodeSYS compiler. See here.

sagatowski commented 4 years ago

Finally implemented. To state this was a major task would be an understatement. See: