Additional test result reported if test calls tmt-report-result

[lukaszachy]

Simple test:

test: tmt-report-result /test/name PASS 

Before ece9db96caa8611f87f43e6e221e9ab793237b01 tmt reported summary: 1 test executed, now it reports summary: 2 tests executed

Milos was super quick to analyze via chat thus storing it here:

        if test.result == 'custom':
            return self.check_custom_results(test, guest)
        if test.framework == 'beakerlib':
            return self.check_beakerlib(test, guest)
        try:
            return self.check_result_file(test, guest)
        except tmt.utils.FileError:
            return self.check_shell(test, guest)

vs today:

        if test.result == 'custom':
            return self.load_custom_results(test, guest)

        return self.load_tmt_report_results(test, guest) \
            + test.test_framework.extract_results(self, test, guest, logger)

Or in other words: Now it is always what tmt-report-result AND framework reports.

Big question is what is right thing to do (and also see rfe for phases - #1354)

[lukaszachy]

Default 'framework' is now 'shell' which means tmt has always some result from framework (base on the exit code). So if some uses 'tmt-report-result' in the test script, they have no way how to "ignore" framework result decision.

[happz]

WRT implementation, there are multiple possible approaches:

• tmt-report-result or framework, but not both
• tmt-report-result plus framework is obviously possible
• tmt-report-result plus framework, but letting framework know tmt-report-result already provided something - and we can even give those results to the framework if needed, and maybe framework would feel like adding something on top of that.

The third one would give us the most options. And it can emulate the first and second options. I can imagine the beakerlib framework inspecting already collected results from tmt-report-result, adding results for phases, not adding results for exit code, etc.

[psss]

So, in order to cover the #1354 use case as well, what about recording all results reported by tmt-report-result and reporting them similarly as we do for custom results, that is each "phase" would be reported and shown in the report output and the overall result would be aggregation of all. The result aggregation would be handled by python code of the framework plugin and responsibility of tmt-report-result would only be to store all results into the file, it would not have to take care of result hierarchy handling. @pfdaly, @happz, @lukaszachy, what do you think?

[happz]

As discussed in the hacking session today, for this particular issue, we'd go with implementation number one, "tmt-report-result or framework, but not both". I'll submit a patch.