byte-physics / igortest

Igor Pro Universal Testing Framework
https://docs.byte-physics.de/igor-unit-testing-framework/
BSD 3-Clause "New" or "Revised" License
7 stars 2 forks source link

Add special Github Actions console output #448

Open Garados007 opened 1 year ago

Garados007 commented 1 year ago

Github Actions allows special markup in the console output and has additional features which helps the developer to find errors and reports fast without looking in the respective log files. I suggest using the following features:

  1. Report of all failed WARN_* as warnings using ::warning::
  2. Report of all failed CHECK_* and REQUIRE_* as errors using ::error::
  3. Group the log lines of each test case, test suite and test run using ::group::, ::endgroup::
  4. Create a nicely human readable Markdown summary in the file referenced by the environment variable GITHUB_STEP_SUMMARY
  5. Add output variables with some metrics like number of test cases, errors, ... by writing to the file referenced by the environment variable GITHUB_OUTPUT. Actions after that can use these variables without reading the JUnit file.

Implementing 1, 2 and 4 can greatly improve the testing experience. 3 is only useful if there are lots of lines between the start and end token (maybe we should use them only for test suites and test runs).

I think this additional output should be always active if the environment variable GITHUB_ACTIONS is set to true which is always the case for Github Actions. There is no need to modify RunTest() as the environment variable can easily be set for debugging purpose and there is no other use case.

Remarks: