search

SatelliteQE / robottelo

Robottelo is a test suite that exercises The Foreman.
GNU General Public License v3.0
61 stars 114 forks source link

better logging of fam failures #16967

Open evgeni opened 3 days ago

evgeni commented 3 days ago

Problem Statement

The output of FAM tests is mangled due to how pytest tries to print the big result object. There are no linebreaks and it's super hard to find the failure.

Additionally the double assert is pointless, as it asserts twice the same thing: the result is successful.

Solution

Stop asserting PASSED in stdout (additionally saves the case where there is the string PASSED in the output for some reason, but the test actually failed). Pass a custom error output to the status == 0 assertion, including stdout, stderr and the status code.

Related Issues

evgeni commented 3 days ago 
trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py
Satellite-QE commented 3 days ago

PRT Result

Build Number: 9416
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 336 warnings in 1228.67s (0:20:28) ===========
evgeni commented 3 days ago 
trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py
Satellite-QE commented 3 days ago

PRT Result

Build Number: 9421
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 328 warnings in 1120.24s (0:18:40) ===========
evgeni commented 3 days ago 
trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py
Satellite-QE commented 3 days ago

PRT Result

Build Number: 9423
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 337 warnings in 1113.46s (0:18:33) ===========
evgeni commented 3 days ago 
trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py
Satellite-QE commented 3 days ago

PRT Result

Build Number: 9424
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 108 failed, 2 passed, 339 warnings in 1260.80s (0:21:00) ===========
evgeni commented 3 days ago 
trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py
Satellite-QE commented 3 days ago

PRT Result

Build Number: 9425
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 16 failed, 94 passed, 341 warnings in 4650.37s (1:17:30) ===========
evgeni commented 2 days ago 
trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py
Satellite-QE commented 2 days ago

PRT Result

Build Number: 9431
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py --external-logging
Test Result : =========== 16 failed, 94 passed, 333 warnings in 4732.56s (1:18:52) ===========
evgeni commented 2 days ago 
trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py -vv
Satellite-QE commented 1 day ago

PRT Result

Build Number: 9445
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py -vv --external-logging
Test Result : =========== 16 failed, 94 passed, 356 warnings in 5042.30s (1:24:02) ===========
evgeni commented 1 day ago 
trigger: test-robottelo
pytest: tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[smart_class_parameter_override_value]
Satellite-QE commented 1 day ago

PRT Result

Build Number: 9447
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/sys/test_fam.py::test_positive_run_modules_and_roles[smart_class_parameter_override_value] --external-logging
Test Result : ================= 1 failed, 11 warnings in 1118.29s (0:18:38) ==================