BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner: Add failure to run Host Test as CI test error.

apop5 commented 1 month ago

Description

When a compiled test encounters an error in running, an error will be output to the log but it will not be logged as a CI failures case.

Adding tests failing to run as a CI failure case by incrementing the failure_count.

[ ] Impacts functionality?
- Functionality - Does the change ultimately impact how firmware functions?
- Examples: Add a new library, publish a new PPI, update an algorithm, ...
[ ] Impacts security?
- Security - Does the change have a direct security impact on an application, flow, or firmware?
- Examples: Crypto algorithm change, buffer overflow fix, parameter validation improvement, ...
[x] Breaking change?
- Breaking change - Will anyone consuming this change experience a break in build or boot behavior?
- Examples: Add a new library class, move a module to a different repo, call a function in a new library class in a pre-existing module, ...
[ ] Includes tests?
- Tests - Does the change include any explicit test code?
- Examples: Unit tests, integration tests, robot tests, ...
[ ] Includes documentation?
- Documentation - Does the change contain explicit documentation additions outside direct code modifications (and comments)?
- Examples: Update readme file, add feature readme file, link to documentation on an a separate Web page, ...

How This Was Tested

In a Repo where there were errors in the host based tests being executed, saw the messages of format in the CI Log, but CI passed:

ERROR - UnitTest Execution Error: <TestName>

After making this change, the UnitTest Execution errors were reported as CI failures.

Integration Instructions

Unit tests that failed to execute will now be reported as CI errors. This will require the UnitTests to be examined as corrected on a case by case basis.

codecov-commenter commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 1.35%. Comparing base (7cf3382) to head (0785913).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## release/202311 #1035 +/- ## ================================================== - Coverage 1.35% 1.35% -0.01% ================================================== Files 1304 1304 Lines 333790 333790 Branches 5103 5103 ================================================== - Hits 4512 4510 -2 - Misses 329195 329197 +2 Partials 83 83 ``` | [Flag](https://app.codecov.io/gh/microsoft/mu_basecore/pull/1035/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) | Coverage Δ | | |---|---|---| | [MdeModulePkg](https://app.codecov.io/gh/microsoft/mu_basecore/pull/1035/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) | `0.68% <ø> (ø)` | | | [MdePkg](https://app.codecov.io/gh/microsoft/mu_basecore/pull/1035/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) | `5.39% <ø> (-0.01%)` | :arrow_down: | | [NetworkPkg](https://app.codecov.io/gh/microsoft/mu_basecore/pull/1035/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) | `0.55% <ø> (ø)` | | | [PolicyServicePkg](https://app.codecov.io/gh/microsoft/mu_basecore/pull/1035/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft) | `30.41% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=microsoft#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Javagedes commented 1 month ago

It should be noted that the scenario in which the error is not truly caught, is when the error happens during the setup before tests start to run. As soon as individual tests run and the the xml is created, the error would be caught. But Aaron has found scenarios where the error and subsequent exiting of the program happens before a single test is run, and we do not catch this.

microsoft / mu_basecore