Generate plain text report with test results

[gctucker]

Now that the pipeline is nearly ready to be deployed on staging, it would be very useful to start making plain text reports with test results. The kind of information would be similar to what is in current KernelCI test reports, for example:

mainline/master baseline: 443 runs, 20 regressions (v5.17-rc1-436-g24f4db1f3a27)

Regressions Summary
-------------------

platform                     | arch  | lab          | compiler | defconfig                    | regressions
-----------------------------+-------+--------------+----------+------------------------------+------------
beagle-xm                    | arm   | lab-baylibre | gcc-10   | omap2plus_defconfig          | 1          
fsl-ls1043a-rdb              | arm64 | lab-nxp      | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
fsl-ls1043a-rdb              | arm64 | lab-nxp      | gcc-10   | defconfig+CON...OMIZE_BASE=y | 1          
imx6ul-14x14-evk             | arm   | lab-nxp      | gcc-10   | imx_v6_v7_defconfig          | 1          
imx8mn-ddr4-evk              | arm64 | lab-nxp      | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
meson-g12a-sei510            | arm64 | lab-baylibre | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
meson-g12a-u200              | arm64 | lab-baylibre | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
meson-g12a-x96-max           | arm64 | lab-baylibre | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
meson-g12b-a311d-khadas-vim3 | arm64 | lab-baylibre | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
meson-g12b-odroid-n2         | arm64 | lab-baylibre | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
meson-gxl-s905x-libretech-cc | arm64 | lab-baylibre | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
meson-sm1-khadas-vim3l       | arm64 | lab-baylibre | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
meson-sm1-sei610             | arm64 | lab-baylibre | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
r8a774c0-ek874               | arm64 | lab-cip      | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
zynqmp-zcu102                | arm64 | lab-cip      | clang-13 | defconfig+CON..._64K_PAGES=y | 1          
zynqmp-zcu102                | arm64 | lab-cip      | clang-13 | defconfig                    | 1          
zynqmp-zcu102                | arm64 | lab-cip      | gcc-10   | defconfig+arm64-chromebook   | 1          
zynqmp-zcu102                | arm64 | lab-cip      | gcc-10   | defconfig+CON...OMIZE_BASE=y | 1          
zynqmp-zcu102                | arm64 | lab-cip      | gcc-10   | defconfig+ima                | 1          
zynqmp-zcu102                | arm64 | lab-cip      | gcc-10   | defconfig+crypto             | 1          

  Details:  https://kernelci.org/test/job/mainline/branch/master/kernel/v5.17-rc1-436-g24f4db1f3a27/plan/baseline/

  Test:     baseline
  Tree:     mainline
  Branch:   master
  Describe: v5.17-rc1-436-g24f4db1f3a27
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
  SHA:      24f4db1f3a2725a6308105081d822b26889e1018 

For this, we would need an event to know when all tests have either completed or timed out. We don't have it yet, so initially we could create a report for every completed test result. Effectively, there would be only one row in the table but the template should be the same.

We also don't have regression information yet in the new KernelCI API, so the report would be showing pass/fail results instead. So a report could look like this:

mainline/master v5.17-rc1-436-g24f4db1f3a27: 1 runs, 0 failures

test             | result
-----------------+-------
check-describe   | Pass

  Tree:     mainline
  Branch:   master
  Describe: v5.17-rc1-436-g24f4db1f3a27
  URL:      https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
  SHA1:     24f4db1f3a2725a6308105081d822b26889e1018 

The main difference here is that the report is for all the test results for a given kernel revision, whereas the real report example is only for a type of tests and shows only regressions. But these are things that can be adjusted later, once we have all the results on a web dashboard and once we have regressions in the KernelCI API.

The report could be generated using a Jinja2 template, all the data in the example should be available via test node objects. Initially, every time a test completes with either a pass or fail status, a report like this one could be generated. It can be simply printed on stdout to make it appear in the docker-compose logs.

Follow-up tasks would then be:

• enable sending it as an email to kernelci-results-staging@groups.io
• listen for events about all tests being complete rather than every test results, to have results grouped in a single report

Other GitHub issues will be created for those, and maybe a few more depending on what we find as we start implementing and using these reports.

[JenySadadia]

Hi @gctucker ,

I have created a list of sub-tasks that need to be done to resolve this issue.

• Add endpoint to publish event in api/main.py in kernelci-api
• Add a wrapper to request publish endpoint in kernelci/data/kernelci_api.py in kernelci-core
• Publish event in runner.py when a test is completed in kernelci-pipeline
• Modify jinja2 template for test report format
• Print test report on stdout
• Enable sending it as an email to kernelci-results-staging@groups.io
• Listen for events about all tests being complete rather than every test result, to have results grouped in a single report

[gctucker]

No need to publish events, that's done automatically when a node is updated. So here's a revised list of sub-tasks:

• Create a client similar to notifier.py but which will only listen for events about completed tests (until we have event filters, we can just discard other ones)
• Every time such event is received, generate a plain-text report using the Node data and a Jinja2 template
• Print the report on stdout

[gctucker]

The list above is for the first step, which I think is what this issue should be about. There will be other things to do after that, but those tasks should be created once the first one has been completed as there may be other things we discover as part of doing it.