search

samba-in-kubernetes / sit-environment

SIT (Samba Integration Testing) Framework
GNU General Public License v3.0
1 stars 7 forks source link

Github records tests as passing even on failure. #51

Closed spuiuk closed 1 year ago

spuiuk commented 1 year ago

We have noticed that centos-ci test run against PRs for sit-test-environment are being recorded as passed even as the console logs show failures. This breaks the CI system and needs to be fixed.

eg: PR - https://github.com/samba-in-kubernetes/sit-test-cases/pull/36 image

This shows gluster tests as passing. But the console logs for gluster tests show the following.

=========================== short test summary info ============================
FAILED testcases/smbtorture/test_smbtorture.py::test_smbtorture[replicate-glusterfs-default-smb2.create]
FAILED testcases/smbtorture/test_smbtorture.py::test_smbtorture[disperse-glusterfs-default-smb2.create]
FAILED testcases/mount/test_mount.py::test_mount[192.168.123.10-disperse-glusterfs-default]
FAILED testcases/mount/test_mount.py::test_mount[192.168.123.11-disperse-glusterfs-default]
================== 4 failed, 52 passed in 8039.14s (2:13:59) ===================
ERROR: InvocationError for command /root/sit-test-cases/.tox/pytest/bin/pytest -v testcases/mount testcases/consistency testcases/smbtorture (exited with code 1)
___________________________________ summary ____________________________________
ERROR:   pytest: commands failed

@anoopcs9 mentioned that this issue is caused by changes introduced in PR #42 and work is being done to change the way these errors are reported.

xhernandez commented 1 year ago

Yes, I'm aware of that. I have some ideas to solve the issue and provide more visibility to failures, but it will require some time. For now, I'll send a PR to cause a failure when a test fails, at least until/if the new approach is implemented and merged.

xhernandez commented 1 year ago

PR #52 should fix the problem for now.