Report overall test as WARN if a subtest reports WARN

juk commented 2 years ago

tmt-report-result: Report overall test as WARN if one of the subtests reports WARN

Below is the expected overall test results matrix

One of the subtests reports "FAIL": report overall test as "FAIL". TMT reports as expected
All subtests report "PASS": report overall test as "PASS". TMT reports as expected
One of the subtests reports "SKIP" and the rest "PASS": report overall test as "PASS". TMT reports as expected
One of the subtests reports "WARN" and the rest "PASS": report overall test as "WARN". TMT does not report as expected

In the last scenario where one subtest reports "WARN" TMT reports overall result as "PASS". The expected overall result would be "WARN"

lukaszachy commented 2 years ago

@pfdaly Could you take a look please?

pfdaly commented 2 years ago

Hi,

Could you please provide some execution output to further clarify this issue for me. I'm not sure of the usage of the term 'Phase' in tmt, do you mean perchance mean 'Node'?

I checked using the smoke node. I edited the smoke.fmf file that exists for this logic such that only /smoke/good and /smoke/warn exist within, so as to check the return code being returned without the inclusion of /smoke/bad, and also changed the return code of the test.sh in the parent report-result directory to 0, so as to check the console report.

[root@pdaly report-result]# tmt -c arch=x86_64 -c distro=centos-stream-8 run --all --verbose provision --how local plan --name /plans/features/basic test --name . /var/tmp/tmt/run-151 Found 1 plan.

/plans/features/basic summary: Basic command line features discover how: fmf directory: /root/auto-kernel/tmt/tmt hash: 13e948a filter: tier: 2 summary: 1 test selected /tests/execute/restraint/report-result provision how: local distro: Red Hat Enterprise Linux 8.5 (Ootpa) kernel: 4.18.0-348.20.1.el8_5.x86_64 summary: 1 guest provisioned prepare how: install summary: Install required packages name: requires package: 3 packages requested beakerlib tmt tmt-provision-container summary: 1 preparation applied execute how: tmt 00:00:15 /tests/execute/restraint/report-result [1/1] summary: 1 test executed report how: display fail /tests/execute/restraint/report-result summary: 1 test failed finish summary: 0 tasks completed

total: 1 test failed [root@pdaly report-result]#

Snippet from: tmt run --last report -fvvv

execute how: tmt order: 50 exit-first: False test: Verify that test results are correctly mapped 00:00:00 /smoke/good [1/2] test: Verify that test results are correctly mapped 00:00:00 /smoke/warn [2/2] summary: 2 tests executed report how: display order: 50 pass /smoke/good output.txt: /var/tmp/tmt/run-151/plan/execute/data/smoke/good/output.txt content: warn /smoke/warn output.txt: /var/tmp/tmt/run-151/plan/execute/data/smoke/warn/output.txt content: summary: 1 test passed and 1 warn finish container: stopped container: removed summary: 0 tasks completed

total: 1 test passed and 1 warn :: [ 21:08:29 ] :: [ FAIL ] :: Command 'tmt run -vvv --remove' (Expected 0, got 1) :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: Duration: 15s :: Assertions: 0 good, 1 bad :: RESULT: FAIL (Test)

As can be seen, the two tests from the smoke node execute on the tmt provisioned container. A return code of 1 is received by tmt after execution on account of the warn, as per the Exit Code specification, this results in the test being marked as a fail in the report.

juk commented 2 years ago

@pfdaly phase is a subcase with result reported through tmt-report-result

Test case we used

TESTARGS=${TESTARGS:-"mixed"}

echo "tmt version: $(rpm -qa | grep tmt)"

phase_test()
{
    echo "Test phase: $1 Result: $2 Result Code: $3"
    report_result "$1" "$2" "$3"
}

echo "Test starts"

case "${TESTARGS}" in
    "mixed")
        phase_test "Regression test 1" "PASS" "0"
        phase_test "Regression test 2" "FAIL" "1"
        phase_test "Regression test 3" "WARN" "2"
        phase_test "Regression test 4" "SKIP" "1"
        phase_test "Regression test 5" "PASS" "0"
        ;;
    "pass")
        phase_test "Regression test 1" "PASS" "0"
        phase_test "Regression test 2" "PASS" "0"
        phase_test "Regression test 3" "PASS" "0"
        phase_test "Regression test 4" "PASS" "0"
        phase_test "Regression test 5" "PASS" "0"
        ;;
    "skip")
        phase_test "Regression test 1" "PASS" "0"
        phase_test "Regression test 2" "PASS" "0"
        phase_test "Regression test 3" "SKIP" "1"
        ;;
    "warn")
        phase_test "Regression test 1" "PASS" "0"
        phase_test "Regression test 2" "WARN" "1"
        phase_test "Regression test 3" "PASS" "0"
        ;;
    *)
        ;;
esac

echo "Test ends"
report_result "report_result test" "PASS" "0"
exit 0

report_result is a wrapper function for:

function report_result {
    rhts-report-result "$1" "$2" "$OUTPUTFILE" "$3"
}

juk commented 2 years ago

└─# tmt  run -d -v -a --environment TESTARGS="warn" provision -h minute -i 1MT-RHEL-<hide> plan --name "/tmt/plans/report_result"
Using tree '/root/Git/kg_test'.
/var/tmp/tmt/run-026
tmt version: 1.15.0 (1ca2898)
Run data not found.
Enabled steps: discover, prepare, finish, provision, report and execute
Found 1 plan.

/tmt/plans/report_result
    Report step always force mode enabled.
    discover
    workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/discover
        how: fmf
        url: https://<hide>/kg_test.git
        Clone 'https://<hide>/kg_test.git' to '/var/tmp/tmt/run-026/tmt/plans/report_result/discover/default/tests'.
        ref: main
        Checkout ref 'main'.
        hash: b993207
        tests: /tmt/issues/report_result
        Check metadata tree in '/var/tmp/tmt/run-026/tmt/plans/report_result/discover/default/tests/'.
        summary: 1 test selected
            /tmt/issues/report_result
    provision
    workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/provision
        how: minute
        image: 1MT-RHEL-<hide>
        flavor: m1.small
        auto-select-network: False
        skip-prereserve: False
        Check for available 1MT images.
        1mt-image: 1MT-RHEL-<hide>
        Trying to get date from the date-service.
        Instance start: 2022-07-20-08-27
        Try to get a prereserved minute machine.
        Prereserved machine result: prereserve-1MT-RHEL-<hide>-44122-2022-07-20-08-18 <hide>
        Rename the machine from 'prereserve-1MT-RHEL-<hide>-44122-2022-07-20-08-18' to 'root-1MT-RHEL-<hide>-140262-2022-07-20-08-27'.
        Change properties of the prereserved machine.
        Execute command 'cat /etc/os-release' on guest '<hide>'.
        Create the master ssh connection: ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i/var/tmp/tmt/run-026/tmt/plans/report_result/provision/default/1minutetip -i/var/tmp/tmt/run-026/tmt/plans/report_result/provision/default/1minutetip-ecdsa -S/run/user/0/tmt/tmpilwojym2 -MNnT root@<hide>
        distro: Red Hat Enterprise Linux 9.1 Beta (Plow)
        Execute command 'uname -r' on guest '<hide>'.
        kernel: <hide>
        summary: 1 guest provisioned
    prepare
    workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/prepare
        Push workdir to guest '<hide>'.
        summary: 0 preparations applied
    execute
    workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/execute
        how: tmt
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/tmt-file-submit' on the guest.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/rstrnt-report-log' on the guest.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/rhts-report-log' on the guest.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-reboot' to '/usr/local/bin/tmt-reboot' on the guest.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-reboot' to '/usr/local/bin/rstrnt-reboot' on the guest.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-reboot' to '/usr/local/bin/rhts-reboot' on the guest.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-report-result' to '/usr/local/bin/tmt-report-result' on the guest.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-report-result' to '/usr/local/bin/rstrnt-report-result' on the guest.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-report-result' to '/usr/local/bin/rhts-report-result' on the guest.
        Push workdir to guest '<hide>'.
        Execute '/tmt/issues/report_result' as a 'shell' test.
        Execute command 'set -eo pipefail; make run' on guest '<hide>'.
            00:00:00 /tmt/issues/report_result [1/1]
        Copy '/var/tmp/tmt/run-026/tmt/plans/report_result/execute/data/tmt/issues/report_result' from the guest to '/'.
        Check result of '/tmt/issues/report_result'.
        Copy '/var/tmp/tmt/run-026/tmt/plans/report_result/data' from the guest to '/'.
        summary: 1 test executed
    report
    workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/report
        how: display
            pass /tmt/issues/report_result
        summary: 1 test passed
    finish
    workdir: /var/tmp/tmt/run-026/tmt/plans/report_result/finish
        guest: removed
        Remove the minute instance 'root-1MT-RHEL-<hide>-140262-2022-07-20-08-27'.
        summary: 0 tasks completed

total: 1 test passed

thrix commented 1 year ago

@pfdaly is that more clear to you now?

teemtee / tmt

Report overall test as WARN if a subtest reports WARN #1353