xenserver / status-report

Program that gathers data for xenserver host diagnostics
GNU Lesser General Public License v2.1
1 stars 9 forks source link

Add pytest-based integration test framework #25

Closed bernhardkaindl closed 10 months ago

bernhardkaindl commented 10 months ago

Add a pytest-based integration test framework, addressing the weaknesses of the initial prototype tests based on shell scripts:

To fix these weaknesses of my early prototype tests, I implemented pytest fixtures which automatically run xen-bugtool in a test container (using Linux namespaces, so it works on any Linux system that would otherwise be capable to run docker, podman, etc), but not requiring any configuration, group or root privileges to install and use.

Add:

An example, why the pytest framework instead of shell scripts is beneficial are for example these undetected test errors in this #22 which is currently in review: https://github.com/xenserver/status-report/actions/runs/6938095957/job/18873299352?pr=22

+ grep -q 'snmpv2cpublic\|0xsnmpv3authenticationkey\|0xsnmpv3privacykey' /code/status-report/.tmp/tests/snmp/tar/snmp_xs_conf.out
grep: /code/status-report/.tmp/tests/snmp/tar/snmp_xs_conf.out: No such file or directory
+ grep -q snmpv2cpublic /code/status-report/.tmp/tests/snmp/tar/snmpd_xs_conf.out
grep: /code/status-report/.tmp/tests/snmp/tar/snmpd_xs_conf.out: No such file or directory
+ grep -q '0xsnmpv3authenticationkey\|0xsnmpv3privacykey' /code/status-report/.tmp/tests/snmp/tar/snmpd_conf.out
grep: /code/status-report/.tmp/tests/snmp/tar/snmpd_conf.out: No such file or directory

With the test implemented in Python, possibilities for silent test errors will be less difficult to spot!

And, with it, we can replace the use of grep with content matching in Python, also using re, when needed.

See the README-pytest.md for further information.