pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.64k stars 2.58k forks source link

`xfail`ed tests are reported as `skipped` on `TestReport` instead of `passed` #12539

Open scratchmex opened 1 week ago

scratchmex commented 1 week ago

problem

when we mark a test as xfail either with strict or not, we expect to fail so in the report we should have it as outcome=passed. in the case of not strict, should always be passed or either xpass. on strict it should be fail if it didn't failed and passed or xfail otherwise

right now the only way of knowing if it was an xfail or skip is to check report.wasxfail attribute

minimal example

# test_example.py
import pytest

@pytest.mark.xfail
def test_fail():
    assert False
# pytest_plugin.py
def pytest_runtest_logreport(report):
    print(report)
python -m pytest -vvvv -s -p pytest_explug
❯ python -m pytest -vvvv -s -p pytest_plugin
================================= test session starts ==================================
platform darwin -- Python 3.11.7, pytest-7.4.2, pluggy-1.3.0 -- /opt/homebrew/opt/python@3.11/bin/python3.11
cachedir: .pytest_cache
rootdir: /private/tmp/testxfail
plugins: anyio-4.0.0
collected 1 item

test_example.py::test_fail 
<TestReport 'test_example.py::test_fail' when='setup' outcome='passed'>
XFAIL
<TestReport 'test_example.py::test_fail' when='call' outcome='skipped'>
<TestReport 'test_example.py::test_fail' when='teardown' outcome='passed'>

reference

https://github.com/pytest-dev/pytest/blob/f74e947c1fdfef238235b7dd18c8fe52108268f2/src/_pytest/skipping.py#L266-L292

in the code above we have some statements marking the outcome as skipped and actually one (I am unaware of that flow) as passed

rep.outcome = "skipped"
# but also
rep.outcome = "passed"