Evaluate test-results, logs a summary and exits process with a exitcode depending on test-results.

densogiaichned commented 3 years ago

This PR enables the evaluation of the test-results internally. Useful if one uses TcUnit-Runner without Jenkins, i.e. GitHub Actions.
No need for additional steps to report via Junit-XML, i.e. KyleAure/junit-report-annotations-action.

2021-05-16 13:10:09 - ... got 28 report lines so far.
2021-05-16 13:10:09 - All results from TcUnit obtained
2021-05-16 13:10:19 - Done collecting TC results
2021-05-16 13:10:19 - Writing xUnit XML file to C:\Users\%USERNAME%\private-actions-runner\_work\TcUnitCiTest\TcUnitCiTest\src\TcUnit_xUnit_results.xml
2021-05-16 13:10:19 - 
Testresults for 1 Testsuite(s) with total 2 Testcase(s):
ID=0, Name="PRG_Tests.fbSum_Tests": Tests=2, Failures=1.
    Testcase "TwoPlusTwoEqualsFour", Assertions=1, ClassName="PRG_Tests.fbSum_Tests", Status="PASS".
    Testcase "ZeroPlusZeroEqualsZero", Assertions=1, ClassName="PRG_Tests.fbSum_Tests", Status="FAIL" Message="The calculation is not correct", Type="INT".

2021-05-16 13:10:19 - Closing the Visual Studio Development Tools Environment (DTE)...
2021-05-16 13:10:39 - Exiting application...
Exit code is 19
Error: Process completed with exit code 19.

sagatowski commented 3 years ago

Thanks a lot for this PR @densogiaichned! I'll do my best to review it during the week!

sagatowski commented 3 years ago

@densogiaichned Sorry I might be slow this late of night, but I thought a GitHub action would normally be defined as any other pipeline (be it Jenkins or whatever), and that you would normally still use the JUnit XML-file output from that as well?

Any documentation update necessary for TcUnit-Runner manual?

sagatowski commented 3 years ago

Also, if you could write an issue (marked as new feature, and link this PR to that) I'd really appreciate that, as I would have something for the release notes. Just describe with a few sentences what this new feature actually does.

densogiaichned commented 3 years ago

Linking to #26

densogiaichned commented 3 years ago

Sorry I might be slow this late of night, but I thought a GitHub action would normally be defined as any other pipeline (be it Jenkins or whatever), and that you would normally still use the JUnit XML-file output from that as well?

Due to the exitcode 0 - RETURN_SUCCESSFULL, a additional step in Github Actions is necessary to evaluate the results, i.e.


      # run TcUnit-Runner  
      - name: Run TcUnit-Runner
        if: success()
        shell: cmd
        run: '"C:\Program Files (x86)\TcUnit-Runner\LaunchTcUnit.bat" -U 5'

      # additional step necessary, to evaluate test-results
      - uses: KyleAure/junit-report-annotations-action@1.5
        if: always()
        with:
          name: "unit tests"
          path: ${{ github.workspace }}/**/TcUnit_xUnit_results.xml

So the result of the workflow depends only on the additional step KyleAure/junit-report-annotations-action, which parses the JUnit XML-file. Without that extra step, your workflow/pipeline would always be successful.
Basically nothing wrong with that, but IMHO it would be better if TcUnit-Runner exits like other test-suites, i.e. VsTest - vstest.console.exe, thus not relying on third party parsers.

Any documentation update necessary for TcUnit-Runner manual?

IHMO not necessary, but i could provide a GitHub action YML-file for documentation or repository, if you want.

sagatowski commented 3 years ago

So the result of the workflow depends only on the additional step KyleAure/junit-report-annotations-action, which parses the JUnit XML-file. Without that extra step, your workflow/pipeline would always be successful. Basically nothing wrong with that, but IMHO it would be better if TcUnit-Runner exits like other test-suites, i.e. VsTest - vstest.console.exe, thus not relying on third party parsers.

Makes sense. Thanks for explanation.

IHMO not necessary, but i could provide a GitHub action YML-file for documentation or repository, if you want.

That would be highly appreciated. Then I could link to that from the documentation website, in case anyone else would want to have a starting point for doing a GitHub action, so this information is not lost.

I'll merge this PR during the week. Please attach the YML-file here as a comment.

densogiaichned commented 3 years ago

TcUnit-Runner with GitHub Action

Runs on a self-hosted runner, i.e. a VM with Windows10 and TwinCAT installed.

Prerequisites:

Add a self hosted runner to a repository
optional with tag, i.e. tcunit If you have multiple runners connected to your repository/organization, a tag like tcunit specifies exactly on which runner a workflow can run.
Create a folder .github/workflows/ in your repository.

In that folder, add a workflow file, i.e. .github/workflows/build_and_test.yml:

# Demo CI workflow with TcUnit-Runner
name: TcUnit-Runner

on:
push:
  branches: [ main ]
pull_request:
  branches: [ main ]

jobs:

tcunit_runner:

  # a self hosted runner, i.e. VM with tag 'tcunit'
  runs-on: [ self-hosted, tcunit ]

  steps:

    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Run TcUnit-Runner
      shell: cmd
      # Start TcUnit-Runner with a timeout of 5 min
      run: '"C:\Program Files (x86)\TcUnit-Runner\LaunchTcUnit.bat" -U 5'

The workflow will be triggered on every pull-request or push to branch main on your self-hosted runner.

optional, reporting annotations:

     # report annotations
     - uses: KyleAure/junit-report-annotations-action@1.5
       if: always()
       with:
         name: results TcUnit tests
         path: ${{ github.workspace }}/**/TcUnit_xUnit_results.xml          
         access-token: ${{ secrets.GITHUB_TOKEN }}

sagatowski commented 3 years ago

@densogiaichned This is gold, thank you!

sagatowski commented 3 years ago

@densogiaichned I did a quick compile, and got lots of errors.

densogiaichned commented 3 years ago

Ohhh , shooot sorry I use VS2019, will fix this!

sagatowski commented 3 years ago

@densogiaichned I assumed so :-) TcUnit is all based on VS2013. Will probably change in the future, but then I will change it for all projects (TcUnit, TcUnit-Runner, TcUnit-Verifier etc).

Btw, I've added your instructions for a GitHub action: https://tcunit.org/tcunit-runner-user-manual/#GitHub_action Added contributions to your name. Thanks for the excellent instructions, I'm sure others will use these!

Looking forward to your second commit!

densogiaichned commented 3 years ago

PR fixes compatibility with VS2013. Expression-bodied property replaced with a function, $-string interpolation replaced with string.Format

sagatowski commented 3 years ago

Did some modifications to the program. Will commit soon. This is the results from running the TcUnit-Verifier_TwinCAT. 2021-05-20_17_14_59.log

sagatowski commented 3 years ago

@densogiaichned Please double-check that the output (file attached above) looks OK for you.

densogiaichned commented 3 years ago

Output looks good! Did you double-check the exit code ? Thanks!

sagatowski commented 3 years ago

@densogiaichned Nope. I'll write a tiny batch-script and try it again. Do you know if there is a way to get the exit code in the command pronpt directly from the TcUnit-Runner.exe?

densogiaichned commented 3 years ago

quick search says that '%ERRORLEVEL%' holds the last return value, i.e.

"C:\Program Files (x86)\TcUnit-Runner\LaunchTcUnit.bat" && echo %ERRORLEVEL%

densogiaichned commented 3 years ago

@sagatowski Its time for a unit-test for the unit-tester.

hehe

sagatowski commented 3 years ago

@densogiaichned There actually already exists a (partial) one :) https://github.com/tcunit/TcUnit/tree/master/TcUnit-Verifier

sagatowski commented 3 years ago

2021-05-20 19:10:57 [INFO ] - Writing xUnit XML file to C:\Code\TcUnit\TcUnit-Verifier\TcUnit-Verifier_TwinCAT\TcUnit_xUnit_results.xml
2021-05-20 19:10:57 [INFO ] - Closing the Visual Studio Development Tools Environment (DTE)...
2021-05-20 19:11:17 [INFO ] - Exiting application...

C:\TcUnit-Runner>echo %ERRORLEVEL%
1

sagatowski commented 3 years ago

Also tested running a successful solution:

C:\TcUnit-Runner>TcUnit-Runner -v "C:\Code\ExampleProjects\SimpleExampleProject\SimpleTcUnitExampleProject.sln"
2021-05-20 19:25:04 [INFO ] - TcUnit-Runner build: 0.9.4.0
2021-05-20 19:25:04 [INFO ] - TcUnit-Runner build date: 2021-05-20
2021-05-20 19:25:04 [INFO ] - Visual Studio solution path: C:\Code\ExampleProjects\SimpleExampleProject\SimpleTcUnitExampleProject.sln
2021-05-20 19:25:04 [INFO ] -
2021-05-20 19:25:04 [INFO ] - In TwinCAT project file, found TwinCAT version 3.1.4022.30
2021-05-20 19:25:04 [INFO ] - In Visual Studio solution file, found visual studio version 12.0
2021-05-20 19:25:04 [INFO ] - Version is pinned: False
2021-05-20 19:25:04 [INFO ] - Trying to load the Visual Studio Development Tools Environment (DTE) version 'VisualStudio.DTE.12.0' ...
2021-05-20 19:25:09 [INFO ] - ...SUCCESSFUL!
2021-05-20 19:25:14 [INFO ] - Using the TwinCAT remote manager to load TwinCAT version '3.1.4022.30'...
2021-05-20 19:25:29 [INFO ] - No task name provided. Assuming only one task exists
2021-05-20 19:25:29 [INFO ] - Found task with name 'PlcTask'
2021-05-20 19:25:43 [INFO ] - Setting target NetId to '127.0.0.1.1.1'
2021-05-20 19:25:44 [INFO ] - Enabling boot project and setting BootProjectAutostart on 127.0.0.1.1.1
2021-05-20 19:26:16 [INFO ] - Waiting for results from TcUnit...
2021-05-20 19:26:26 [INFO ] - ... got 25 report lines so far.
2021-05-20 19:26:26 [INFO ] - All results from TcUnit obtained
2021-05-20 19:26:36 [INFO ] - Done collecting TC results
2021-05-20 19:26:36 [INFO ] -
Test results for 1 test suite(s) with total 2 test(s):
ID=0, Name="PRG_TEST.fbSum_Test": Tests=2, Failures=0.
        Test case "TwoPlusTwoEqualsFour", Assertions=1, ClassName="PRG_TEST.fbSum_Test", Status="PASS".
        Test case "ZeroPlusZeroEqualsZero", Assertions=1, ClassName="PRG_TEST.fbSum_Test", Status="PASS".

2021-05-20 19:26:36 [INFO ] - Writing xUnit XML file to C:\Code\ExampleProjects\SimpleExampleProject\TcUnit_xUnit_results.xml
2021-05-20 19:26:36 [INFO ] - Closing the Visual Studio Development Tools Environment (DTE)...
2021-05-20 19:26:56 [INFO ] - Exiting application...

C:\TcUnit-Runner>echo %ERRORLEVEL%
0

All looks good!

densogiaichned commented 3 years ago

TcUnit-Runner with Azure Pipelines

Runs on a self-hosted agent, i.e. a VM with Windows 10 and TwinCAT installed.

Prerequisites:

Integrate Your GitHub Projects With Azure Pipelines
Add a self-hosted agent to Azure Pipelines
and name it, i.e. tcunit.

Create a Azure Pipeline
This will add a azure-pipelines.yml in your repository, copy and paste the following content:

# Demo CI workflow with TcUnit-Runner
name: TcUnit-Runner  

trigger:
- main
pr:
- main  

pool: 'tcunit'    # self-hosted agent  

steps:  

- checkout: self
  displayName: Checkout repository  

- task: CmdLine@2
  displayName: Run PLC unit-tests with TcUnit-Runner 
  condition: succeeded()
  inputs:
      failOnStderr: true
      # Start TcUnit-Runner with a timeout of 5 min
      script: '"C:\Program Files (x86)\TcUnit-Runner\LaunchTcUnit.bat" -U 5'

- task: PublishTestResults@2
  displayName: Publish TcUnit results
  condition: or(succeeded(),failed())     
  inputs:
  testResultsFormat: 'JUnit'
  testResultsFiles: '**/TcUnit_xUnit_results.xml'

Save and run your new Pipeline.
The workflow will be triggered on every pull-request or push to branch main on your self-hosted agent.

sagatowski commented 3 years ago

The workflow will be triggered on every pull-request or push to branch main on your self-hosted agent.

Thank you very much!

tcunit / TcUnit-Runner