search

teemtee / tmt

Test Management Tool
MIT License
81 stars 122 forks source link

tmt run execute -vvvddd is not equal to tmt run -vvvddd execute #1550

Open sopos opened 2 years ago

sopos commented 2 years ago 
$ tmt run --last plans --default execute -vvvddd -f --interactive --how tmt tests --name .
/var/tmp/tmt/rhel-9

/plans/default
    Clean up workdir '/var/tmp/tmt/rhel-9/plans/default/execute'.
    Read file '/var/tmp/tmt/rhel-9/plans/default/execute/step.yaml'.
    Step data not found.
    Read file '/var/tmp/tmt/rhel-9/plans/default/execute/results.yaml'.
    Test results not found.
    Using the 'ExecuteInternal' plugin for the 'tmt' method.
    status: todo
    Write file '/var/tmp/tmt/rhel-9/plans/default/execute/step.yaml'.
    Write file '/var/tmp/tmt/rhel-9/plans/default/execute/results.yaml'.
    execute
    workdir: /var/tmp/tmt/rhel-9/plans/default/execute
        how: tmt
        order: 50
        exit-first: False
        Write file '/var/tmp/tmt/rhel-9/plans/default/execute/data/Sanity/stdin-stdout-stderr-not-a-terminal/metadata.yaml'.
            test: if stdin/stdout/stderr is not a terminal SELinux should not interfere with sudo
        Execute '/Sanity/stdin-stdout-stderr-not-a-terminal' as a 'beakerlib' test.
        Use workdir '/var/tmp/tmt/rhel-9/plans/default/discover/default-0/tests/Sanity/stdin-stdout-stderr-not-a-terminal'.
Warning: Identity file None not accessible: No such file or directory.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Setup
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
$ tmt run -vvvddd --last plans --default execute -f --interactive --how tmt tests --name .
...

/plans/default
...
    execute
    workdir: /var/tmp/tmt/rhel-9/plans/default/execute
        how: tmt
        order: 50
        exit-first: False
        Write file '/var/tmp/tmt/rhel-9/plans/default/execute/data/Sanity/stdin-stdout-stderr-not-a-terminal/metadata.yaml'.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-abort' to '/usr/local/bin/tmt-abort' on the guest.
        Create the master ssh connection: ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -iNone -S/run/user/1000/tmt/tmpxtyf8u9t -MNnT root@sopos-rhel9-brq.usersys.redhat.co
m
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-abort root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/tmt-abort
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-abort' to '/usr/local/bin/rstrnt-abort' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-abort root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/rstrnt-abort
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-abort' to '/usr/local/bin/rhts-abort' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-abort root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/rhts-abort
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/tmt-file-submit' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-file-submit root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/tmt-file-submit
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/rstrnt-report-log' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-file-submit root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/rstrnt-report-log
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/rhts-submit-log' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-file-submit root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/rhts-submit-log
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-file-submit' to '/usr/local/bin/rhts_submit_log' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-file-submit root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/rhts_submit_log
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-reboot' to '/usr/local/bin/tmt-reboot' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-reboot root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/tmt-reboot
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-reboot' to '/usr/local/bin/rstrnt-reboot' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-reboot root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/rstrnt-reboot
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-reboot' to '/usr/local/bin/rhts-reboot' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-reboot root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/rhts-reboot
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-report-result' to '/usr/local/bin/tmt-report-result' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-report-result root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/tmt-report-result
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-report-result' to '/usr/local/bin/rstrnt-report-result' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-report-result root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/rstrnt-report-result
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Copy '/usr/lib/python3.10/site-packages/tmt/steps/execute/scripts/tmt-report-result' to '/usr/local/bin/rhts-report-result' on the guest.
        Run command: rsync -p --chmod=755 -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /usr/lib/python3.10/site-packages
/tmt/steps/execute/scripts/tmt-report-result root@sopos-rhel9-brq.usersys.redhat.com:/usr/local/bin/rhts-report-result
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
        Push workdir to guest 'sopos-rhel9-brq.usersys.redhat.com'.
        Run command: rsync -R -r -z --links --safe-links --delete -e 'ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -i'"'"''"'"' -S/run/user/1000/tmt/tmpxtyf8u9t' /var/tmp/
tmt/rhel-9/plans/default root@sopos-rhel9-brq.usersys.redhat.com:/
        err: Warning: Identity file -S/run/user/1000/tmt/tmpxtyf8u9t not accessible: No such file or directory.
        err: Warning: Permanently added 'sopos-rhel9-brq.usersys.redhat.com' (ED25519) to the list of known hosts.
            test: if stdin/stdout/stderr is not a terminal SELinux should not interfere with sudo
        Execute '/Sanity/stdin-stdout-stderr-not-a-terminal' as a 'beakerlib' test.
        Use workdir '/var/tmp/tmt/rhel-9/plans/default/discover/default-0/tests/Sanity/stdin-stdout-stderr-not-a-terminal'.
        Execute command './runtest.sh' on guest 'sopos-rhel9-brq.usersys.redhat.com'.
        Run command: ssh -oForwardX11=no -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oIdentitiesOnly=yes -iNone -S/run/user/1000/tmt/tmpxtyf8u9t -t root@sopos-rhel9-brq.usersys.redhat.com 'export TMT_PLAN_DATA=
/var/tmp/tmt/rhel-9/plans/default/data TMT_TREE=/var/tmp/tmt/rhel-9/plans/default/tree TMT_TEST_DATA=/var/tmp/tmt/rhel-9/plans/default/execute/data/Sanity/stdin-stdout-stderr-not-a-terminal/data TMT_REBOOT_REQUEST=/var/tmp/tm
t/rhel-9/plans/default/execute/data/Sanity/stdin-stdout-stderr-not-a-terminal/data/reboot-request TMT_REBOOT_COUNT=0 REBOOTCOUNT=0 RSTRNT_REBOOTCOUNT=0 BEAKERLIB_DIR=/var/tmp/tmt/rhel-9/plans/default/execute/data/Sanity/stdin
-stdout-stderr-not-a-terminal BEAKERLIB_COMMAND_SUBMIT_LOG='"'"'bash /usr/local/bin/tmt-file-submit'"'"'; cd /var/tmp/tmt/rhel-9/plans/default/discover/default-0/tests/Sanity/stdin-stdout-stderr-not-a-terminal; ./runtest.sh'
Warning: Identity file None not accessible: No such file or directory.

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
::   Setup
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
pvalena commented 1 year ago

Isn't this expected behaviour, as you set verbosity for individual steps.

e.g. $ tmt run execute -vvv report -v

pvalena commented 1 year ago

https://github.com/teemtee/tmt/issues/509

idorax commented 1 year ago

@sopos, according to #509 and PR https://github.com/teemtee/tmt/pull/616, looks tmt run execute -vvvddd and tmt run -vvvddd execute should behave differently as expected, can we close this issue?

sopos commented 1 year ago

I do not agree with closing. The verbosity should be still propagated from the global to local.

My understanding is that the global setting should be inherited unless a local is specified.

There is one little catch: how to disregard the verbosity for one step? Implementing -s|--silent might be a solution.

The same applies to the debug.

idorax commented 1 year ago

Hi @sopos,

sopos commented 1 year ago

I would say so. That's my take when @psss explained it to me some time ago - the sub-commands should inherit these options in a cascade.

idorax commented 1 year ago

That's my take when @psss explained it to me some time ago - the sub-commands should inherit these options in a cascade.

Hi @psss, would you help to explain what the differences among tmt run execute -vvvddd, tmt run -vvvddd execute and tmt -dddvvv run execute?

From my understanding, tmt -dddvvv run execute means -dddvvv is global as:

$ tmt --help
Usage: tmt [OPTIONS] COMMAND [ARGS]...

but I don't understand the expected behaviours of both tmt run -vvvddd execute and tmt run execute -vvvddd, would you please help to clarify about sub-commands should inherit these options in a cascade? thanks!

lukaszachy commented 1 year ago

@idorax It means that subcommand should user verbosity/debug level from main command unless it is specified explicitly.

command tmt run discover execute
tmt -vvvddd run discover execute v=3 d=3 v=3 d=3 v=3 d=3 v=3 d=3
tmt run -vvvddd discover execute v=0 d=0 v=3 d=3 v=3 d=3 v=3 d=3
tmt run discover -vvvddd execute v=0 d=0 v=0 d=0 v=3 d=3 v=0 d=0
tmt -v run -dd discover -vvd execute v=1 d=0 v=1 d=2 v=2 d=1 v=1 d=2

So for 'execute' step all three tmt run execute -vvvddd, tmt run -vvvddd execute and tmt -dddvvv run execute has same verbosity/debug values. What would differ are other steps (if they were run)

The reported issue shows that even though execute -vvvddd was used, output is less verbose than it should be.

happz commented 1 year ago

Very helpful table! What is the expected outcome of the following, decreasing verbosity? tmt -vvvddd run -vvdd discover -vd execute

sopos commented 1 year ago

not really, execute should inherit from run, i.e. it would effectively be run -vvdd

lukaszachy commented 1 year ago

What is the expected outcome of the following, decreasing verbosity? tmt -vvvddd run -vvdd discover -vd execute

As @sopos wrote: "steps" are siblings (not parent-child) so verbosity of discover doesn't affect execute thus the verbosity of the execute step should be inherited from it's parent which is run.

lukaszachy commented 1 year ago

We are missing 'silent' or decrease verbosity level. Or is there a way from tmt -v run -dd execute to run execute with v=0 d=1?

sopos commented 1 year ago

I'm afraid there's no such a way. Therefore I suggested to implement -s|--silent to be able to achieve that. But The priority is really low as one usually wants to debug more rather than less.