KStocky / ShaderTestFramework

An automation testing framework for testing shader code
MIT License
20 stars 1 forks source link

Create Custom Test Reporter and Job Summary action #113

Open KStocky opened 3 months ago

KStocky commented 3 months ago

Is your feature request related to a problem? Please describe. With #112 we got some basic stat tracking and reporting. However, there are a couple of things missing

  1. We can't actually, output our stats to any reporter
  2. Even if we could output it to a reporter there don't seem to be any github actions that would be able to publish those stats as a job summary.

So the stats are kinda pointless right now except for debugging purposes. I would rather be able to report them and display them as part of the job summary.

Describe the solution you'd like We need a custom catch2 reporter - The JUnit format does support extra parameters that can be attached to test cases. So we could take catch2's junit reporter and add on this extra functionality (and perhaps push it back to them)

We need a custom job summary creator - Job summaries are produced just by piping the standard output of an application into an environment variable. This is a great primer on bash for this. So we would need to create a program/script that will take in any number of JUnit XML files, parse them, and output them to std::cout in markdown format to generate the job summary.

These are two large tasks. But it would be cool.