Better `run.sh` tests execution

[pvalena]

Adresses https://github.com/psss/tmt/issues/93.

[pvalena]

Environment is actually logged with the command execution, if specified. Note also this line where it is defined.

Anyway, to make it apparent, I've added separate entries for all 3 "additonal test entries": duration, path, environment.

Or did you mean other than modified env entries? We're using: --ignore-environment -i -- start with empty environment for env.

[pvalena]

@lukaszachy PTAL, and please see the added note.

[packit-as-a-service[bot]]

Congratulations! One of the builds has completed. :champagne:

You can install the built RPMs by following these steps:

• sudo yum install -y dnf-plugins-core on RHEL 8
• sudo dnf install -y dnf-plugins-core on Fedora
• dnf copr enable packit/psss-tmt-99
• And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

[psss]

Thanks for the improvement, @pvalena. I have two comments:

I've tried to run one of the tmt tests with added environment definition:

tmt run -d plan --name basic test --name ls

but I've noticed that the test fails because it cannot find beakerlib journal:

> > path: cd '/var/tmp/tmt/run-031/plans/basic/discover/one/tests/tests/ls' && 
> > duration: timeout '5m' 
> > environment: env -i x=1 y=2 z=3 
> > beakerlib execute: cd '/var/tmp/tmt/run-031/plans/basic/discover/one/tests/tests/ls' && env -i x=1 y=2 z=3 timeout '5m' ./test.sh
grep: journal.txt: No such file or directory
Error: Result not found

This seems to be related to the modified environment, probably missing the BEAKERLIB_DIR variable? Could you please fix that so that we can verify the functionality?

And second: What about doing something like this before executing the test script?

set > environment.txt

Being able to check all variables defined in the environment could be useful and I think that was the original use case reported by @lukaszachy in #93.

[pvalena]

Oh, would you rather have env output, or set output? Does it make sense to have it logged (duplicitly) to stdout/stderr also?

I'll look into the error, the behaviour was actually not changed with this PR, so it's broken at this point (I've missed propagating the ENV var for beakerlib to the right place). Should be easy fix.

[psss]

Oh, would you rather have env output, or set output? Does it make sense to have it logged (duplicitly) to stdout/stderr also?

I think storing the env to a file in the workdir should be enough. @lukaszachy, what do you think?

I'll look into the error, the behaviour was actually not changed with this PR, so it's broken at this point (I've missed propagating the ENV var for beakerlib to the right place). Should be easy fix.

Thanks!

[psss]

@lukaszachy, how the environment details should be stored? See comments above for details.

[lukaszachy]

@lukaszachy, how the environment details should be stored? See comments above for details.

I usually record output of declare -p so I can see exact definition of variable, however set should be enough too. Having separate file with the environment will be better, as env is quite huge.

[psss]

I think it would be good to address the execution problem as part of this pull request too, so that we can actually test/verify it is working as expected. @pvalena, could you have a look at this soon? This is blocking test execution under modified environment. Thanks.

[pvalena]

I believe I've adressed the issues.

[pvalena]

I did however encounter errors, during installation, or make test:

(project_venv) $  project_venv/bin/pip install --upgrade '.[all]'
 [ . . . ]
Installing collected packages: libvirt-python, testcloud, tmt
  Running setup.py install for libvirt-python ... error
    ERROR: Command errored out with exit status 1:
     command: /home/lpcs/lpcsf-new/test/tmt/project_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-57hdtqcn/libvir
t-python/setup.py'"'"'; __file__='"'"'/tmp/pip-install-57hdtqcn/libvirt-python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().re
place('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-5so_mb4a/install-record.txt --singl
e-version-externally-managed --compile --install-headers /home/lpcs/lpcsf-new/test/tmt/project_venv/include/site/python3.8/libvirt-python
         cwd: /tmp/pip-install-57hdtqcn/libvirt-python/
    Complete output (13 lines):
    Package libvirt was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libvirt.pc'                                                                                                     to the PKG_CONFIG_PATH environment variable                                                                                                                      Package 'libvirt', required by 'virtual:world', not found                                                                                                        Package libvirt was not found in the pkg-config search path.
    Perhaps you should add the directory containing `libvirt.pc'
    to the PKG_CONFIG_PATH environment variable
    Package 'libvirt', required by 'virtual:world', not found
    running install
    running build
    /usr/bin/pkg-config --print-errors --atleast-version=0.9.11 libvirt
    Package 'libvirt' was not found
    error: command '/usr/bin/pkg-config' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/lpcs/lpcsf-new/test/tmt/project_venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"
'/tmp/pip-install-57hdtqcn/libvirt-python/setup.py'"'"'; __file__='"'"'/tmp/pip-install-57hdtqcn/libvirt-python/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"',
 open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-5so_mb4a/install-record.txt --single-version-externally-managed --compile --install-headers /home/lpcs/lpcsf-new/test/tmt/project_venv/include/site/python3.8/libvirt-python Check the logs for full command output.                                                                                                                WARNING: You are using pip version 19.2.3, however version 20.0.2 is available.                                                                                  You should consider upgrading via the 'pip install --upgrade pip' command.

I did install dependencies using dnf- testcloud, vagrant-libvirt, python-libvirt ... but I still get the error above. (I do have even tmt-all installed on my rawhide machine.)

[pvalena]

I do have an error too, with run:

(project_venv) ➜  tmt git:(run.sh) tmt run -d
Traceback (most recent call last):
  File "/home/lpcs/lpcsf-new/test/tmt/project_venv/bin/tmt", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/lpcs/lpcsf-new/test/tmt/bin/tmt", line 7, in <module>
    import tmt.cli
  File "/home/lpcs/lpcsf-new/test/tmt/tmt/__init__.py", line 3, in <module>
    from tmt.base import Tree, Test, Plan, Story, Run
  File "/home/lpcs/lpcsf-new/test/tmt/tmt/base.py", line 15, in <module>
    import tmt.steps.provision
  File "/home/lpcs/lpcsf-new/test/tmt/tmt/steps/provision/__init__.py", line 11, in <module>
    from tmt.steps.provision import vagrant, localhost, podman, testcloud
  File "/home/lpcs/lpcsf-new/test/tmt/tmt/steps/provision/testcloud.py", line 7, in <module>
    import testcloud.image
ModuleNotFoundError: No module named 'testcloud'

[thrix]

I do have an error too, with run:

(project_venv) ➜  tmt git:(run.sh) tmt run -d
Traceback (most recent call last):
  File "/home/lpcs/lpcsf-new/test/tmt/project_venv/bin/tmt", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/lpcs/lpcsf-new/test/tmt/bin/tmt", line 7, in <module>
    import tmt.cli
  File "/home/lpcs/lpcsf-new/test/tmt/tmt/__init__.py", line 3, in <module>
    from tmt.base import Tree, Test, Plan, Story, Run
  File "/home/lpcs/lpcsf-new/test/tmt/tmt/base.py", line 15, in <module>
    import tmt.steps.provision
  File "/home/lpcs/lpcsf-new/test/tmt/tmt/steps/provision/__init__.py", line 11, in <module>
    from tmt.steps.provision import vagrant, localhost, podman, testcloud
  File "/home/lpcs/lpcsf-new/test/tmt/tmt/steps/provision/testcloud.py", line 7, in <module>
    import testcloud.image
ModuleNotFoundError: No module named 'testcloud'

@pvalena testcloud provisioner was merged and bring additional deps. I guess you rebased on top of master branch. Now install all required deps via pip install -e .

[psss]

I tried to run tmt run -d for this change and I see many errors:

 > /one/tests/docs:
 >     duration: 5m
 >     path: /one/tests/tests/docs
 >     test: ./test.sh
 > > path: /var/tmp/tmt/run-289/plans/basic/discover/one/tests/tests/docs
 > > environment: 
 > > duration: timeout '5m' 
 > > command: ./test.sh
 > > type: beakerlib
grep: journal.txt: No such file or directory
Error: Result not found
 > > out.log:
touch: cannot touch '/journal.meta': Permission denied

Similar errors appear when running in a container. Using virtual I see also error messages:

> /help/main:
>     path: .
>     test: tmt --help
> > path: /var/tmp/tmt/run-290/plans/helps/discover
> > environment: 
> > duration: 
> > command: tmt --help
> > type: shell
> > out.log:
env: ‘’: No such file or directory
> > exitcode.log:

This does not seem to be working.

[pvalena]

This does not seem to be working.

Yes, I know, I did not get around to debug it :( . I'm getting the same error.

[psss]

Thanks for the update. Seems to be working now. Only the environment.txt file is created under discover instead of execute. It should go into the same folder where the results, journal and output are stored.

[pvalena]

Thanks for the update. Seems to be working now. Only the environment.txt file is created under discover instead of execute. It should go into the same folder where the results, journal and output are stored.

Oh, that's a bug. It should go under every test path. So maybe it's because your path was . ? It should never be that ... :))

[pvalena]

Ok, so one part fixed:

$ cd /var/tmp/tmt/run-066 && find -type f -name environment.txt -exec bash -c "set -x; cat {}" \;
+ cat ./plans/helps/execute/help/smoke/environment.txt
PWD=/var/tmp/tmt/run-066/plans/helps/discover/one/tests/tests/shell
SHLVL=1
_=/usr/bin/env
+ cat ./plans/helps/execute/help/plan/environment.txt
PWD=/var/tmp/tmt/run-066/plans/helps/discover                                                                                                                                       SHLVL=1                                                                                                                                                                             _=/usr/bin/env                                                                                                                                                                      + cat ./plans/helps/execute/help/main/environment.txt                                                                                                                               PWD=/var/tmp/tmt/run-066/plans/helps/discover                                                                                                                                       SHLVL=1                                                                                                                                                                             _=/usr/bin/env                                                                                                                                                                      + cat ./plans/helps/execute/help/test/environment.txt                                                                                                                               PWD=/var/tmp/tmt/run-066/plans/helps/discover
SHLVL=1
_=/usr/bin/env
+ cat ./plans/basic/execute/one/tests/ls/environment.txt
PWD=/var/tmp/tmt/run-066/plans/basic/discover/one/tests/tests/ls
BEAKERLIB_DIR=/var/tmp/tmt/run-066/plans/basic/execute/one/tests/ls
SHLVL=1
_=/usr/bin/env
+ cat ./plans/basic/execute/one/tests/docs/environment.txt
PWD=/var/tmp/tmt/run-066/plans/basic/discover/one/tests/tests/docs
BEAKERLIB_DIR=/var/tmp/tmt/run-066/plans/basic/execute/one/tests/docs
SHLVL=1
_=/usr/bin/env
+ cat ./plans/smoke/execute/script-00/environment.txt
PWD=/var/tmp/tmt/run-066/plans/smoke/discover
SHLVL=1
_=/usr/bin/env

[pvalena]

Oh, that's a bug. It should go under every test path. So maybe it's because your path was . ? It should never be that ... :))

Actually, the runtime may be anywhere the user wants. Specifying . is good with me. I'll make it default also, so you don't have to input it in tests.yaml.

[psss]

Thanks for the update. With the latest change works fine. Tested with local, virtual and container.

[pvalena]

Thanks for the update. With the latest change works fine. Tested with local, virtual and container.

YAW, thanks for testing!