Add support for test phases

[psss]

As discussed on the stakeholder meeting, there are use cases related to tracking individual results of test "phases". These can be created using one of the following ways:

• defining a beakerlib phase using rlPhaseStart and rlPhaseEnd
• explicitly calling rstrnt-report-result to submit a subresult

When run in Beaker, the rlPhaseEnd command called during a beakerlib test execution actually calls rlReport which uses BEAKERLIB_COMMAND_REPORT_RESULT command to report the result and, if I'm not mistaken, this defaults to using rstrnt-report-result. @sopos, please correct me if I'm wrong.

The effect is that each beakerlib phase and rstrnt-report-result call create a "subresult" which is shown in the Beaker web interface under the actual test. The proposal is to add a similar support for subresults to tmt.

Here are some more ideas caught from the brainstorm:

• We could inject a custom script using BEAKERLIB_COMMAND_REPORT_RESULT and perform necessary actions on the guest
• The result specification could be extended to include a list of subresults (which would be similar to the check attribute)
• It would be good to report detected avc failures for each test phase separately
• Report plugins would have to invent the proper mapping to e.g. junit format as the spec does not know subresults

All in all it seems this would bring a substantial level of complexity (currently the test granularity was simply defined by the test metadata definition for each test) so we should discuss this well before jumping into the implementation.

Base Work

• spec — https://github.com/teemtee/tmt/pull/2933
• class — https://github.com/teemtee/tmt/pull/3094
• https://github.com/teemtee/tmt/pull/3106

Report Plugins

• display — https://github.com/teemtee/tmt/pull/3107
• html — https://github.com/teemtee/tmt/pull/3095
• junit
  • https://github.com/teemtee/tmt/pull/3150
  • https://github.com/teemtee/tmt/pull/3177
• polarion — https://github.com/teemtee/tmt/pull/3166

[juk]

@psss I quickly checked a junit format and it seems like there is infrastructure to support subtests/phases classname - could be used for a parent test name name - could be used for a subtest/phase

https://github.com/testmoapp/junitxml

[kkaarreell]

According to https://www.ibm.com/docs/en/developer-for-zos/14.2?topic=formats-junit-xml-format <testsuite> has child elements <testcase>, not <testsuite> again. We should be clear in which artifacts these phases should appear and how those artifacts are going to be used. Do we know if the proposed XML with nested <testsuite> elements would be compatible with ReportPortal for example?

[kkaarreell]

FYI, I just tried to import https://github.com/testmoapp/junitxml/blob/main/examples/junit-basic.xml to RP and it went well, creating nested structures for testsuites. So at least from the RP perspective we may be fine.

[lukaszachy]

Note: Beakerlib executes BEAKERLIB_COMMAND_REPORT_RESULT only if TESTID is set (https://github.com/beakerlib/beakerlib/blob/4321c54e2e99f74b91c9376aa4324dafd0300c3d/src/testing.sh#L1075) - we don't set it in tmt yet.

[seberm]

Just FYI, I was experimenting with a nested structure of <testsuite> using a junit-xml but the library does not support it. There are unmerged patches that add the support for nested <testsuite> structures (quickly tested - they seem to work fine), but the library is outdated and unsupported (the last commit was 4 years ago).

I have been searching for suitable alternative libraries, but so far, I haven't found any. Any ideas?

[thrix]

Doing a fallback to own implementation should not be that hard, note that there is a nice way via attr/cattrs we use in Testing Farm and @janhavlin was working on it and could help porting it to tmt:

https://gitlab.com/testing-farm/gluetool-modules/-/tree/main/gluetool_modules_framework/libs/results?ref_type=heads

[sopos]

Note breakers is able to produce the journal.xml in the breaker format as well as in the xunit. However, I was recently contacted by someone who was having an issue where the respective xml python module was missing which causes just an error and the xml file not being created. It might be good if tmt was able to parse the journal.meta, i.e. to use the beakerlib's code to transform it (using the journalling.py).

[seberm]

Doing a fallback to own implementation should not be that hard, note that there is a nice way via attr/cattrs we use in Testing Farm and @janhavlin was working on it and could help porting it to tmt:

https://gitlab.com/testing-farm/gluetool-modules/-/tree/main/gluetool_modules_framework/libs/results?ref_type=heads

Regarding our implementation of the JUnit XML library, I was thinking that we might be able to directly use the junit_xml (approximately just 400-lines of code + unit tests).

We could apply the necessary patches for "test suite inside a test suite", clean up the code, remove Python2 support, and add the type annotations.

Then it should very easily integrate into the TMT. The license for junit_xml is the same as for TMT - the MIT license:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

We would likely be fine by simply adding the copyright notice to the library file header and referencing to the original project repository.

What are your thoughts on this? Do you think this could be a viable solution?