Closed lukaszachy closed 10 months ago
Default 'framework' is now 'shell' which means tmt has always some result from framework (base on the exit code). So if some uses 'tmt-report-result' in the test script, they have no way how to "ignore" framework result decision.
WRT implementation, there are multiple possible approaches:
tmt-report-result
or framework, but not bothtmt-report-result
plus framework is obviously possibletmt-report-result
plus framework, but letting framework know tmt-report-result
already provided something - and we can even give those results to the framework if needed, and maybe framework would feel like adding something on top of that.The third one would give us the most options. And it can emulate the first and second options. I can imagine the beakerlib
framework inspecting already collected results from tmt-report-result
, adding results for phases, not adding results for exit code, etc.
So, in order to cover the #1354 use case as well, what about recording all results reported by tmt-report-result
and reporting them similarly as we do for custom results, that is each "phase" would be reported and shown in the report output and the overall result would be aggregation of all. The result aggregation would be handled by python code of the framework
plugin and responsibility of tmt-report-result
would only be to store all results into the file, it would not have to take care of result hierarchy handling. @pfdaly, @happz, @lukaszachy, what do you think?
As discussed in the hacking session today, for this particular issue, we'd go with implementation number one, "tmt-report-result or framework, but not both". I'll submit a patch.
Simple test:
Before ece9db96caa8611f87f43e6e221e9ab793237b01 tmt reported
summary: 1 test executed
, now it reportssummary: 2 tests executed
Milos was super quick to analyze via chat thus storing it here:
vs today:
Or in other words: Now it is always what tmt-report-result AND framework reports.
Big question is what is right thing to do (and also see rfe for phases - #1354)