search

astraw38 / pytest-assume

A pytest plugin that allows multiple failures per test
MIT License
66 stars 16 forks source link

Does not work when combined with @flaky #14

Closed Chuckv closed 5 years ago

Chuckv commented 5 years ago

This plugin does not work for tests which are being re-run on assert failure via the Flaky plugin.

Install flaky plugin and then try the following

from flaky import flaky

@flaky()
def test_error(self):
    pytest.assume(False)

The result that I got:

===Flaky Test Report===

test_error passed 1 out of the required 1 times. Success!

===End Flaky Test Report===

========================= no tests ran in 3.06 seconds =========================
astraw38 commented 5 years ago

Edit: I'm looking into it more closely. There's some shenanigans with how flaky looks at whether tests pass/fail.

astraw38 commented 5 years ago

As an alternative, pytest-rerunfailures definitely works with it. It'll take some time to sort out what flaky is doing though.

astraw38 commented 5 years ago

After further review, flaky will only rerun if a test raises an exception. Most of the time this is indeed the case, but for pytest.assume we don't want to raise an exception during the test -- instead we complete the test & set the report to failed afterwards.

I'll look into whether this is possible to be fixed... but no guarantees.

Chuckv commented 5 years ago

at my workplace we have several hundred tests that use @flaky decorators (mostly I think due to sauce and networking issues) so it would be really nice if this could work with that.. I don't see us wanting to change over to pytest-rerunfailures unless it also offers advantages over flaky

astraw38 commented 5 years ago

Can you give the branch https://github.com/astraw38/pytest-assume/tree/use_new_hook here a trial to see if it works for your setup?

I did my best to verify locally w/ flaky, but it's got some bleedthrough with pytester (and doesn't fully seem to fully disable itself even with `-p no:flaky).

Chuckv commented 5 years ago

how do I make use of that branch? (sorry, not very familiar with pytest plugins) is there a way to install it with pip? (I'm a bit of a newbie when it comes to python)

astraw38 commented 5 years ago

pip install git+https://github.com/astraw38/pytest-assume@use_new_hook

It should install pytest-assume version 2.0.0 (which you can use to verify via pip show pytest-assume)

Chuckv commented 5 years ago

Oh thanks, so I can use pip, cool. will try to have a look shortly..

astraw38 commented 5 years ago

Closing -- merged in #16 which should fix this.