flaky should retry also on code exception?

[andreabisello]

i have understand (but maybe i'm wrong) that flaky will retry failed test due assertion check, but not failing test due code exception.

for example,

i'm testing an API call.

this api call returns a json.

this json normally has some properties ,

but sometime it can return a timeout,

so the try to access this json propertie will raise an exception, that will not makes the test fail.

Sure, i can assert the existence of the property , making the test fail, so flaky will retry, but when i use flaky, i use because i cannot have the complete control of what is making the test fail. In this case, my call was wrapped in a methot that assert the http status code is 200, but in this api implementation, the api returns always 200, even on exception or timeout.

if "flaky" can catch exception raising a "pytest.fail" instead of an exception, it will also retries flaky test for code excetion

what do you think?

thanks

[Jeff-Meadows]

I think flaky does retry on assertions or on other types of exceptions. If it isn't doing so, please file an issue with some code that reproduces the behavior.

[andreabisello]

@Jeff-Meadows

the problem is using the fixtures

e.g.

this will works

from flaky import flaky
import pytest
from pytest import fixture
import random

@fixture()
def not_failing_fixture():
    return random.randrange(1,10)

@flaky(10,1)
def test_using_not_failing_fixture(not_failing_fixture):
    print("obtaining " + str(not_failing_fixture))
    assert not_failing_fixture > 8

it produce :

test_flaky.py::test_using_not_failing_fixture PASSED                     [100%]obtaining 7
obtaining 8
obtaining 2
obtaining 4
obtaining 3
obtaining 9

===Flaky Test Report===

test_using_not_failing_fixture failed (9 runs remaining out of 10).
    <class 'AssertionError'>
    assert left > right failed.
  Showing split diff:

  left:  7
  right: 8

    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:19>]
test_using_not_failing_fixture failed (8 runs remaining out of 10).
    <class 'AssertionError'>
    assert left > right failed.
  Showing split diff:

  left:  8
  right: 8

    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:19>]
test_using_not_failing_fixture failed (7 runs remaining out of 10).
    <class 'AssertionError'>
    assert left > right failed.
  Showing split diff:

  left:  2
  right: 8

    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:19>]
test_using_not_failing_fixture failed (6 runs remaining out of 10).
    <class 'AssertionError'>
    assert left > right failed.
  Showing split diff:

  left:  4
  right: 8

    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:19>]
test_using_not_failing_fixture failed (5 runs remaining out of 10).
    <class 'AssertionError'>
    assert left > right failed.
  Showing split diff:

  left:  3
  right: 8

    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:19>]
test_using_not_failing_fixture passed 1 out of the required 1 times. Success!

===End Flaky Test Report===

this will also works

def randomly_obtain_exception():
    random_number = random.randrange(0, 1)
    return 1 / random_number

@flaky(5,1)
def test_using_random_exception_no_fixture():
    num = randomly_obtain_exception()
    assert num > 0.5

it produce

================================== FAILURES ===================================
___________________ test_using_random_exception_no_fixture ____________________

    @flaky(5,1)
    def test_using_random_exception_no_fixture():
>       num = randomly_obtain_exception()

test_flaky.py:42: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def randomly_obtain_exception():
        random_number = random.randrange(0, 1)
>       return 1 / random_number
E       ZeroDivisionError: division by zero

test_flaky.py:38: ZeroDivisionError
===Flaky Test Report===

test_using_random_exception_no_fixture failed (4 runs remaining out of 5).
    <class 'ZeroDivisionError'>
    division by zero
    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:42>, <TracebackEntry C:\src\nso-testsuite\test_flaky.py:38>]
test_using_random_exception_no_fixture failed (3 runs remaining out of 5).
    <class 'ZeroDivisionError'>
    division by zero
    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:42>, <TracebackEntry C:\src\nso-testsuite\test_flaky.py:38>]
test_using_random_exception_no_fixture failed (2 runs remaining out of 5).
    <class 'ZeroDivisionError'>
    division by zero
    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:42>, <TracebackEntry C:\src\nso-testsuite\test_flaky.py:38>]
test_using_random_exception_no_fixture failed (1 runs remaining out of 5).
    <class 'ZeroDivisionError'>
    division by zero
    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:42>, <TracebackEntry C:\src\nso-testsuite\test_flaky.py:38>]
test_using_random_exception_no_fixture failed; it passed 0 out of the required 1 times.
    <class 'ZeroDivisionError'>
    division by zero
    [<TracebackEntry C:\src\nso-testsuite\test_flaky.py:42>, <TracebackEntry C:\src\nso-testsuite\test_flaky.py:38>]

===End Flaky Test Report===

BUT , if assertion inside fixture is failing, the flaky will not retry

@fixture()
def failing_fixture_by_assertion():
    random_number = random.randrange(1,10)
    assert random_number > 5
    return random_number

@flaky(5,1)
def test_using_failing_fixture_by_assertion(failing_fixture_by_assertion):
    print("obtaining " + str(failing_fixture_by_assertion))
    assert failing_fixture_by_assertion > 5

it produce :

test_flaky.py::test_using_failing_fixture_by_assertion ERROR             [100%]
test setup failed
@fixture()
    def failing_fixture_by_assertion():
        random_number = random.randrange(1,10)
>       assert random_number > 5
E       assert left > right failed.
E         Showing split diff:
E         
E         left:  2
E         right: 5

test_flaky.py:9: AssertionError

=================================== ERRORS ====================================
__________ ERROR at setup of test_using_failing_fixture_by_assertion __________

    @fixture()
    def failing_fixture_by_assertion():
        random_number = random.randrange(1,10)
>       assert random_number > 5
E       assert left > right failed.
E         Showing split diff:
E         
E         left:  2
E         right: 5

test_flaky.py:9: AssertionError

OR

if fixture fails for code excetion,

flaky will not retry

@fixture()
def failing_fixture_by_code_exception():
    random_number = random.randrange(0, 1)
    return 1/random_number

@flaky(5,1)
def test_using_failing_fixture_by_code_exception(failing_fixture_by_code_exception):
    print("obtaining " + str(failing_fixture_by_assertion))
    assert failing_fixture_by_assertion > 0.5

it produce

test_flaky.py::test_using_failing_fixture_by_code_exception ERROR        [100%]
test setup failed
@fixture()
    def failing_fixture_by_code_exception():
        random_number = random.randrange(0, 1)
>       return 1/random_number
E       ZeroDivisionError: division by zero

test_flaky.py:15: ZeroDivisionError