search

pytest-dev / pytest-forked

extracted --boxed from pytest-xdist to ensure backward compat
MIT License
62 stars 21 forks source link

[FR] Integrate `xfail` #33

Closed webknjaz closed 4 years ago

webknjaz commented 4 years ago

I have a test that SEGFAULTs under random envs (usually only in one or two jobs in the matrix). So I decided to use forked to isolate it and marked it with xfail as follows:

@pytest.mark.xfail(
    reason='This test causes SEGFAULT, flakily.',
    strict=False,
)
@pytest.mark.forked
def test_exec_command(ssh_channel):
    """Test getting the output of a remotely executed command."""
    ...

But, despite my expectations, when it crashes, pytest shows it as a failure and not as XFAIL.

Could you please clarify whether this behavior is intended or is it a bug? (looks like a bug to me)

Am I doing anything wrong? Am I missing something here?

_(Ref: https://github.com/ansible/pylibssh/runs/728770734?check_suite_focus=true#step:9:120)_

RonnyPfannschmidt commented 4 years ago

Segfaults are currently not integrated

webknjaz commented 4 years ago

Is it hard to integrate them?

RonnyPfannschmidt commented 4 years ago

i have no idea, i haven't worked on it and i don't plan to maintain/work on pytest-forked i only took it out of xdist to enable dropping the boxed feature from xdist itself

webknjaz commented 4 years ago

@RonnyPfannschmidt where can I see how xfail is processed? I'm trying to figure out how things work in src/_pytest/mark/structures.py but it seems like it's probably not the place where I can hack into...

I want to contribute but struggling to find a starting point.

RonnyPfannschmidt commented 4 years ago

I vaguely recall that xfail is processed in the skipping plugin

webknjaz commented 4 years ago

So it turned out that for xfail to work, I need to set outcome = 'skipped' and wasxfail = 'the reason message' attrs on the report object.

webknjaz commented 4 years ago

So it turned out that for xfail to work, I need to set outcome = 'skipped' and wasxfail = 'the reason message' attrs on the report object.

Opened an issue about this https://github.com/pytest-dev/pytest/issues/7327