Closed densogiaichned closed 3 years ago
Thanks a lot for this PR @densogiaichned! I'll do my best to review it during the week!
@densogiaichned Sorry I might be slow this late of night, but I thought a GitHub action would normally be defined as any other pipeline (be it Jenkins or whatever), and that you would normally still use the JUnit XML-file output from that as well?
Any documentation update necessary for TcUnit-Runner manual?
Also, if you could write an issue (marked as new feature, and link this PR to that) I'd really appreciate that, as I would have something for the release notes. Just describe with a few sentences what this new feature actually does.
Linking to #26
Sorry I might be slow this late of night, but I thought a GitHub action would normally be defined as any other pipeline (be it Jenkins or whatever), and that you would normally still use the JUnit XML-file output from that as well?
Due to the exitcode 0 - RETURN_SUCCESSFULL
, a additional step in Github Actions is necessary to evaluate the results, i.e.
# run TcUnit-Runner
- name: Run TcUnit-Runner
if: success()
shell: cmd
run: '"C:\Program Files (x86)\TcUnit-Runner\LaunchTcUnit.bat" -U 5'
# additional step necessary, to evaluate test-results
- uses: KyleAure/junit-report-annotations-action@1.5
if: always()
with:
name: "unit tests"
path: ${{ github.workspace }}/**/TcUnit_xUnit_results.xml
So the result of the workflow depends only on the additional step KyleAure/junit-report-annotations-action
, which parses the JUnit XML-file. Without that extra step, your workflow/pipeline would always be successful.
Basically nothing wrong with that, but IMHO it would be better if TcUnit-Runner
exits like other test-suites, i.e. VsTest - vstest.console.exe
, thus not relying on third party parsers.
Any documentation update necessary for TcUnit-Runner manual?
IHMO not necessary, but i could provide a GitHub action YML-file for documentation or repository, if you want.
So the result of the workflow depends only on the additional step
KyleAure/junit-report-annotations-action
, which parses the JUnit XML-file. Without that extra step, your workflow/pipeline would always be successful. Basically nothing wrong with that, but IMHO it would be better ifTcUnit-Runner
exits like other test-suites, i.e.VsTest - vstest.console.exe
, thus not relying on third party parsers.
Makes sense. Thanks for explanation.
IHMO not necessary, but i could provide a GitHub action YML-file for documentation or repository, if you want.
That would be highly appreciated. Then I could link to that from the documentation website, in case anyone else would want to have a starting point for doing a GitHub action, so this information is not lost.
I'll merge this PR during the week. Please attach the YML-file here as a comment.
Prerequisites:
tcunit
If you have multiple runners connected to your repository/organization, a tag like tcunit
specifies exactly on which runner a workflow can run..github/workflows/
in your repository.In that folder, add a workflow file, i.e. .github/workflows/build_and_test.yml
:
# Demo CI workflow with TcUnit-Runner
name: TcUnit-Runner
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
tcunit_runner:
# a self hosted runner, i.e. VM with tag 'tcunit'
runs-on: [ self-hosted, tcunit ]
steps:
- name: Checkout repository
uses: actions/checkout@v2
- name: Run TcUnit-Runner
shell: cmd
# Start TcUnit-Runner with a timeout of 5 min
run: '"C:\Program Files (x86)\TcUnit-Runner\LaunchTcUnit.bat" -U 5'
main
on your self-hosted runner. # report annotations
- uses: KyleAure/junit-report-annotations-action@1.5
if: always()
with:
name: results TcUnit tests
path: ${{ github.workspace }}/**/TcUnit_xUnit_results.xml
access-token: ${{ secrets.GITHUB_TOKEN }}
@densogiaichned This is gold, thank you!
@densogiaichned I did a quick compile, and got lots of errors.
Ohhh , shooot sorry I use VS2019, will fix this!
@densogiaichned I assumed so :-) TcUnit is all based on VS2013. Will probably change in the future, but then I will change it for all projects (TcUnit, TcUnit-Runner, TcUnit-Verifier etc).
Btw, I've added your instructions for a GitHub action: https://tcunit.org/tcunit-runner-user-manual/#GitHub_action Added contributions to your name. Thanks for the excellent instructions, I'm sure others will use these!
Looking forward to your second commit!
PR fixes compatibility with VS2013.
Expression-bodied property replaced with a function, $-string interpolation replaced with string.Format
Did some modifications to the program. Will commit soon. This is the results from running the TcUnit-Verifier_TwinCAT. 2021-05-20_17_14_59.log
@densogiaichned Please double-check that the output (file attached above) looks OK for you.
Output looks good! Did you double-check the exit code ? Thanks!
@densogiaichned Nope. I'll write a tiny batch-script and try it again. Do you know if there is a way to get the exit code in the command pronpt directly from the TcUnit-Runner.exe?
quick search says that '%ERRORLEVEL%' holds the last return value, i.e.
"C:\Program Files (x86)\TcUnit-Runner\LaunchTcUnit.bat" && echo %ERRORLEVEL%
@sagatowski Its time for a unit-test for the unit-tester.
hehe
@densogiaichned There actually already exists a (partial) one :) https://github.com/tcunit/TcUnit/tree/master/TcUnit-Verifier
2021-05-20 19:10:57 [INFO ] - Writing xUnit XML file to C:\Code\TcUnit\TcUnit-Verifier\TcUnit-Verifier_TwinCAT\TcUnit_xUnit_results.xml
2021-05-20 19:10:57 [INFO ] - Closing the Visual Studio Development Tools Environment (DTE)...
2021-05-20 19:11:17 [INFO ] - Exiting application...
C:\TcUnit-Runner>echo %ERRORLEVEL%
1
Also tested running a successful solution:
C:\TcUnit-Runner>TcUnit-Runner -v "C:\Code\ExampleProjects\SimpleExampleProject\SimpleTcUnitExampleProject.sln"
2021-05-20 19:25:04 [INFO ] - TcUnit-Runner build: 0.9.4.0
2021-05-20 19:25:04 [INFO ] - TcUnit-Runner build date: 2021-05-20
2021-05-20 19:25:04 [INFO ] - Visual Studio solution path: C:\Code\ExampleProjects\SimpleExampleProject\SimpleTcUnitExampleProject.sln
2021-05-20 19:25:04 [INFO ] -
2021-05-20 19:25:04 [INFO ] - In TwinCAT project file, found TwinCAT version 3.1.4022.30
2021-05-20 19:25:04 [INFO ] - In Visual Studio solution file, found visual studio version 12.0
2021-05-20 19:25:04 [INFO ] - Version is pinned: False
2021-05-20 19:25:04 [INFO ] - Trying to load the Visual Studio Development Tools Environment (DTE) version 'VisualStudio.DTE.12.0' ...
2021-05-20 19:25:09 [INFO ] - ...SUCCESSFUL!
2021-05-20 19:25:14 [INFO ] - Using the TwinCAT remote manager to load TwinCAT version '3.1.4022.30'...
2021-05-20 19:25:29 [INFO ] - No task name provided. Assuming only one task exists
2021-05-20 19:25:29 [INFO ] - Found task with name 'PlcTask'
2021-05-20 19:25:43 [INFO ] - Setting target NetId to '127.0.0.1.1.1'
2021-05-20 19:25:44 [INFO ] - Enabling boot project and setting BootProjectAutostart on 127.0.0.1.1.1
2021-05-20 19:26:16 [INFO ] - Waiting for results from TcUnit...
2021-05-20 19:26:26 [INFO ] - ... got 25 report lines so far.
2021-05-20 19:26:26 [INFO ] - All results from TcUnit obtained
2021-05-20 19:26:36 [INFO ] - Done collecting TC results
2021-05-20 19:26:36 [INFO ] -
Test results for 1 test suite(s) with total 2 test(s):
ID=0, Name="PRG_TEST.fbSum_Test": Tests=2, Failures=0.
Test case "TwoPlusTwoEqualsFour", Assertions=1, ClassName="PRG_TEST.fbSum_Test", Status="PASS".
Test case "ZeroPlusZeroEqualsZero", Assertions=1, ClassName="PRG_TEST.fbSum_Test", Status="PASS".
2021-05-20 19:26:36 [INFO ] - Writing xUnit XML file to C:\Code\ExampleProjects\SimpleExampleProject\TcUnit_xUnit_results.xml
2021-05-20 19:26:36 [INFO ] - Closing the Visual Studio Development Tools Environment (DTE)...
2021-05-20 19:26:56 [INFO ] - Exiting application...
C:\TcUnit-Runner>echo %ERRORLEVEL%
0
All looks good!
Prerequisites:
tcunit
. Create a Azure Pipeline
This will add a azure-pipelines.yml
in your repository, copy and paste the following content:
# Demo CI workflow with TcUnit-Runner
name: TcUnit-Runner
trigger:
- main
pr:
- main
pool: 'tcunit' # self-hosted agent
steps:
- checkout: self
displayName: Checkout repository
- task: CmdLine@2
displayName: Run PLC unit-tests with TcUnit-Runner
condition: succeeded()
inputs:
failOnStderr: true
# Start TcUnit-Runner with a timeout of 5 min
script: '"C:\Program Files (x86)\TcUnit-Runner\LaunchTcUnit.bat" -U 5'
- task: PublishTestResults@2
displayName: Publish TcUnit results
condition: or(succeeded(),failed())
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: '**/TcUnit_xUnit_results.xml'
main
on your self-hosted agent.
- The workflow will be triggered on every pull-request or push to branch
main
on your self-hosted agent.
This PR enables the evaluation of the test-results internally. Useful if one uses TcUnit-Runner without Jenkins, i.e. GitHub Actions.
No need for additional steps to report via Junit-XML, i.e.
KyleAure/junit-report-annotations-action
.