Add a pytest-based integration test framework, addressing the weaknesses of the initial prototype tests based on shell scripts:
The initial shell-based tests were quick to create, but only meant as an early prototype:
They need root capabilities through GitHub CI or act (using a local docker or podman as backend container runtime) to run. Such container runtime is not installed on some servers, so it was not easy to check them.
It was easy to overlook errors in the shell scripts, leading to skipped checks, e.g. from the current SNMP feature PR, where this happens (but test does not fail because of it, see below)
To fix these weaknesses of my early prototype tests, I implemented pytest fixtures which automatically run xen-bugtool in a test container (using Linux namespaces, so it works on any Linux system that would otherwise be capable to run docker, podman, etc), but not requiring any configuration, group or root privileges to install and use.
Add:
The test framework using automatically always-used pytest fixtures
A comprehensive README-pytest.md
An example test case:
Check the output expected of xen-bugtool -y --entries=xenserver-logs (including the /etc/systemd.tar archive) that we just merged recently.
More tests to be added and converted with other PRs
+ grep -q 'snmpv2cpublic\|0xsnmpv3authenticationkey\|0xsnmpv3privacykey' /code/status-report/.tmp/tests/snmp/tar/snmp_xs_conf.out
grep: /code/status-report/.tmp/tests/snmp/tar/snmp_xs_conf.out: No such file or directory
+ grep -q snmpv2cpublic /code/status-report/.tmp/tests/snmp/tar/snmpd_xs_conf.out
grep: /code/status-report/.tmp/tests/snmp/tar/snmpd_xs_conf.out: No such file or directory
+ grep -q '0xsnmpv3authenticationkey\|0xsnmpv3privacykey' /code/status-report/.tmp/tests/snmp/tar/snmpd_conf.out
grep: /code/status-report/.tmp/tests/snmp/tar/snmpd_conf.out: No such file or directory
With the test implemented in Python, possibilities for silent test errors will be less difficult to spot!
And, with it, we can replace the use of grep with content matching in Python, also using re, when needed.
Add a
pytest
-based integration test framework, addressing the weaknesses of the initial prototype tests based on shell scripts:act
(using a localdocker
orpodman
as backend container runtime) to run. Such container runtime is not installed on some servers, so it was not easy to check them.To fix these weaknesses of my early prototype tests, I implemented
pytest
fixtures which automatically runxen-bugtool
in a test container (using Linux namespaces, so it works on any Linux system that would otherwise be capable to rundocker
,podman
, etc), but not requiring any configuration, group or root privileges to install and use.Add:
pytest
fixturesREADME-pytest.md
xen-bugtool -y --entries=xenserver-logs
(including the/etc/systemd.tar
archive) that we just merged recently.An example, why the
pytest
framework instead of shell scripts is beneficial are for example these undetected test errors in this #22 which is currently in review: https://github.com/xenserver/status-report/actions/runs/6938095957/job/18873299352?pr=22With the test implemented in Python, possibilities for silent test errors will be less difficult to spot!
And, with it, we can replace the use of
grep
with content matching in Python, also usingre
, when needed.See the
README-pytest.md
for further information.