Closed juk closed 10 months ago
@pfdaly Could you take a look please?
Hi,
Could you please provide some execution output to further clarify this issue for me. I'm not sure of the usage of the term 'Phase' in tmt, do you mean perchance mean 'Node'?
I checked using the smoke node. I edited the smoke.fmf file that exists for this logic such that only /smoke/good and /smoke/warn exist within, so as to check the return code being returned without the inclusion of /smoke/bad, and also changed the return code of the test.sh in the parent report-result directory to 0, so as to check the console report.
[root@pdaly report-result]# tmt -c arch=x86_64 -c distro=centos-stream-8 run --all --verbose provision --how local plan --name /plans/features/basic test --name . /var/tmp/tmt/run-151 Found 1 plan.
/plans/features/basic summary: Basic command line features discover how: fmf directory: /root/auto-kernel/tmt/tmt hash: 13e948a filter: tier: 2 summary: 1 test selected /tests/execute/restraint/report-result provision how: local distro: Red Hat Enterprise Linux 8.5 (Ootpa) kernel: 4.18.0-348.20.1.el8_5.x86_64 summary: 1 guest provisioned prepare how: install summary: Install required packages name: requires package: 3 packages requested beakerlib tmt tmt-provision-container summary: 1 preparation applied execute how: tmt 00:00:15 /tests/execute/restraint/report-result [1/1] summary: 1 test executed report how: display fail /tests/execute/restraint/report-result summary: 1 test failed finish summary: 0 tasks completed
total: 1 test failed [root@pdaly report-result]#
Snippet from: tmt run --last report -fvvv
execute how: tmt order: 50 exit-first: False test: Verify that test results are correctly mapped 00:00:00 /smoke/good [1/2] test: Verify that test results are correctly mapped 00:00:00 /smoke/warn [2/2] summary: 2 tests executed report how: display order: 50 pass /smoke/good output.txt: /var/tmp/tmt/run-151/plan/execute/data/smoke/good/output.txt content: warn /smoke/warn output.txt: /var/tmp/tmt/run-151/plan/execute/data/smoke/warn/output.txt content: summary: 1 test passed and 1 warn finish container: stopped container: removed summary: 0 tasks completed
total: 1 test passed and 1 warn :: [ 21:08:29 ] :: [ FAIL ] :: Command 'tmt run -vvv --remove' (Expected 0, got 1) :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: Duration: 15s :: Assertions: 0 good, 1 bad :: RESULT: FAIL (Test)
As can be seen, the two tests from the smoke node execute on the tmt provisioned container. A return code of 1 is received by tmt after execution on account of the warn, as per the Exit Code specification, this results in the test being marked as a fail in the report.
@pfdaly phase is a subcase with result reported through tmt-report-result
Test case we used
TESTARGS=${TESTARGS:-"mixed"}
echo "tmt version: $(rpm -qa | grep tmt)"
phase_test()
{
echo "Test phase: $1 Result: $2 Result Code: $3"
report_result "$1" "$2" "$3"
}
echo "Test starts"
case "${TESTARGS}" in
"mixed")
phase_test "Regression test 1" "PASS" "0"
phase_test "Regression test 2" "FAIL" "1"
phase_test "Regression test 3" "WARN" "2"
phase_test "Regression test 4" "SKIP" "1"
phase_test "Regression test 5" "PASS" "0"
;;
"pass")
phase_test "Regression test 1" "PASS" "0"
phase_test "Regression test 2" "PASS" "0"
phase_test "Regression test 3" "PASS" "0"
phase_test "Regression test 4" "PASS" "0"
phase_test "Regression test 5" "PASS" "0"
;;
"skip")
phase_test "Regression test 1" "PASS" "0"
phase_test "Regression test 2" "PASS" "0"
phase_test "Regression test 3" "SKIP" "1"
;;
"warn")
phase_test "Regression test 1" "PASS" "0"
phase_test "Regression test 2" "WARN" "1"
phase_test "Regression test 3" "PASS" "0"
;;
*)
;;
esac
echo "Test ends"
report_result "report_result test" "PASS" "0"
exit 0
report_result is a wrapper function for:
function report_result {
rhts-report-result "$1" "$2" "$OUTPUTFILE" "$3"
}
└─# tmt run -d -v -a --environment TESTARGS="warn" provision -h minute -i 1MT-RHEL-<hide> plan --name "/tmt/plans/report_result"
Using tree '/root/Git/kg_test'.
/var/tmp/tmt/run-026
tmt version: 1.15.0 (1ca2898)
Run data not found.
Enabled steps: discover, prepare, finish, provision, report and execute
Found 1 plan.
/tmt/plans/report_result
Report step always force mode enabled.
discover
workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/discover
how: fmf
url: https://<hide>/kg_test.git
Clone 'https://<hide>/kg_test.git' to '/var/tmp/tmt/run-026/tmt/plans/report_result/discover/default/tests'.
ref: main
Checkout ref 'main'.
hash: b993207
tests: /tmt/issues/report_result
Check metadata tree in '/var/tmp/tmt/run-026/tmt/plans/report_result/discover/default/tests/'.
summary: 1 test selected
/tmt/issues/report_result
provision
workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/provision
how: minute
image: 1MT-RHEL-<hide>
flavor: m1.small
auto-select-network: False
skip-prereserve: False
Check for available 1MT images.
1mt-image: 1MT-RHEL-<hide>
Trying to get date from the date-service.
Instance start: 2022-07-20-08-27
Try to get a prereserved minute machine.
Prereserved machine result: prereserve-1MT-RHEL-<hide>-44122-2022-07-20-08-18 <hide>
Rename the machine from 'prereserve-1MT-RHEL-<hide>-44122-2022-07-20-08-18' to 'root-1MT-RHEL-<hide>-140262-2022-07-20-08-27'.
Change properties of the prereserved machine.
Execute command 'cat /etc/os-release' on guest '<hide>'.
Create the master ssh connection: ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i/var/tmp/tmt/run-026/tmt/plans/report_result/provision/default/1minutetip -i/var/tmp/tmt/run-026/tmt/plans/report_result/provision/default/1minutetip-ecdsa -S/run/user/0/tmt/tmpilwojym2 -MNnT root@<hide>
distro: Red Hat Enterprise Linux 9.1 Beta (Plow)
Execute command 'uname -r' on guest '<hide>'.
kernel: <hide>
summary: 1 guest provisioned
prepare
workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/prepare
Push workdir to guest '<hide>'.
summary: 0 preparations applied
execute
workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/execute
how: tmt
Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/tmt-file-submit' on the guest.
Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/rstrnt-report-log' on the guest.
Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/rhts-report-log' on the guest.
Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-reboot' to '/usr/local/bin/tmt-reboot' on the guest.
Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-reboot' to '/usr/local/bin/rstrnt-reboot' on the guest.
Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-reboot' to '/usr/local/bin/rhts-reboot' on the guest.
Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-report-result' to '/usr/local/bin/tmt-report-result' on the guest.
Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-report-result' to '/usr/local/bin/rstrnt-report-result' on the guest.
Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-report-result' to '/usr/local/bin/rhts-report-result' on the guest.
Push workdir to guest '<hide>'.
Execute '/tmt/issues/report_result' as a 'shell' test.
Execute command 'set -eo pipefail; make run' on guest '<hide>'.
00:00:00 /tmt/issues/report_result [1/1]
Copy '/var/tmp/tmt/run-026/tmt/plans/report_result/execute/data/tmt/issues/report_result' from the guest to '/'.
Check result of '/tmt/issues/report_result'.
Copy '/var/tmp/tmt/run-026/tmt/plans/report_result/data' from the guest to '/'.
summary: 1 test executed
report
workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/report
how: display
pass /tmt/issues/report_result
summary: 1 test passed
finish
workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/finish
guest: removed
Remove the minute instance 'root-1MT-RHEL-<hide>-140262-2022-07-20-08-27'.
summary: 0 tasks completed
total: 1 test passed
@pfdaly is that more clear to you now?
tmt-report-result
: Report overall test as WARN if one of the subtests reports WARNBelow is the expected overall test results matrix
In the last scenario where one subtest reports "WARN" TMT reports overall result as "PASS". The expected overall result would be "WARN"