asam-ev / qc-framework

Mozilla Public License 2.0
13 stars 6 forks source link

Module to output results as GitHub workflow commands #92

Open ClemensLinnhoff opened 2 weeks ago

ClemensLinnhoff commented 2 weeks ago

Is your feature request related to a problem? Please describe. Currently the output of the checks is bundled into a results XML file. But when the checker framework is running in a CI pipeline, there needs to be a module interpreting the results, so the pipeline can fail with error messages, give warnings or pass.

Describe the solution you'd like For GitHub there is a syntax to output warnings, errors and infos, see this example.

My first question is: Is something like this planned in the scope of the currently running project? If this is not planned within the project, I would volunteer to implement it for GitHub.

What would be the best way to implement this? Would it make sense to implement it as a checker bundle, that receives the concatenated results of previous checker bundles? Is something like this possible in the current framework?

Or does is make sense to implement this as a standalone application separate from the QC framework?

hoangtungdinh commented 2 weeks ago

Hi @ClemensLinnhoff, this is indeed an interesting and useful feature to consider. Currently we have no plans to implement it in the project.

IMHO, given the architecture of the framework, this feature can be implemented as a report module. A report module takes the result XML file (in xqar format) and converts the result to other desired formats, such as the GitHub WorkFlow Command format.

An example report module that converts the xqar result files to a text file can be found here. Although the example report module is implemented in C++, it is also possible to implement a new text report module in Python using the Python Base Library.

You can use the demo pipeline to obtain the example xqar result file and the output of the report module.

Note that instructions for integrating a new Python-based text report module into the framework will be available in the future.

I'm looking forward to hearing from others about this feature.