search

box / flaky

Plugin for nose or pytest that automatically reruns flaky tests.
Apache License 2.0
377 stars 60 forks source link

flaky fails with pytest==8.1.0 #198

Closed skshetry closed 6 months ago

skshetry commented 6 months ago 
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.12.2/x64/bin/pytest", line 8, in <module>
    sys.exit(console_main())
             ^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 195, in console_main
    code = main()
           ^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 153, in main
    config = _prepareconfig(args, plugins)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 335, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 138, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/helpconfig.py", line 105, in pytest_cmdline_parse
    config = yield
             ^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1141, in pytest_cmdline_parse
    self.parse(args)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1490, in parse
    self._preparse(args, addopts=addopts)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1377, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 414, in load_setuptools_entrypoints
    plugin = ep.load()
             ^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/assertion/rewrite.py", line 178, in exec_module
    exec(co, module.__dict__)
  File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/flaky/flaky_pytest_plugin.py", line 5, in <module>
    from _pytest.runner import call_runtest_hook  # pylint:disable=import-error
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ImportError: cannot import name 'call_runtest_hook' from '_pytest.runner' (/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py)

call_runtest_hook no longer exists and has been inlined. See https://github.com/pytest-dev/pytest/pull/11854.

The-Compiler commented 6 months ago

This project seems to be pretty dead (last release in 2020, last commit half a year ago, https://github.com/box/flaky/issues/192 with no reaction).

You might want to look into migrating to pytest-rerunfailures which is actively maintained and under the pytest-dev community namespace.

qci-amos commented 6 months ago

flaky should either be fixed or it should be removed from the list here: https://docs.pytest.org/en/stable/explanation/flaky.html#plugins

kloczek commented 6 months ago

Just retested 3.8.0 with pytest 8.1.1 and still I see 5 failing units

Here is pytest output: ```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages + /usr/bin/pytest -ra -m 'not network' ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 rootdir: /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0 plugins: flaky-3.7.0 collected 77 items test/test_flaky_decorator.py ... [ 3%] test/test_flaky_plugin.py FF. [ 7%] test/test_multiprocess_string_io.py FF [ 10%] test/test_pytest/pytest_generate_example/test_pytest_generate_example.py [ 11%] .. [ 12%] test/test_pytest/test_flaky_pytest_plugin.py ........................... [ 48%] ..................... [ 75%] test/test_pytest/test_pytest_example.py .xx.x...s......s [ 94%] test/test_pytest/test_pytest_options_example.py F.. [ 98%] test/test_pytest/test_pytester_plugin.py . [100%] =================================== FAILURES =================================== _ TestFlakyPlugin.test_flaky_plugin_handles_non_ascii_byte_string_in_exception _ self = def test_flaky_plugin_handles_non_ascii_byte_string_in_exception(self): mock_method_name = 'my_method' mock_exception = 'ńőń ȁŝćȉȉ ŝƭȕƒƒ'.encode('utf-16') mock_message = 'information about retries' # pylint:disable=protected-access > self._flaky_plugin._log_test_failure( mock_method_name, (ValueError.__name__, mock_exception, ''), mock_message, ) test/test_flaky_plugin.py:38: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../../BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/_flaky_plugin.py:38: in _log_test_failure formatted_exception_info = ''.join(format_exception(*err)).replace('\n', '\n\t').rstrip() /usr/lib64/python3.9/traceback.py:120: in format_exception return list(TracebackException( /usr/lib/python3.9/site-packages/exceptiongroup/_formatting.py:96: in __init__ self.stack = traceback.StackSummary.extract( /usr/lib64/python3.9/traceback.py:347: in extract for f, lineno in frame_gen: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tb = '' def walk_tb(tb): """Walk a traceback yielding the frame and line number for each frame. This will follow tb.tb_next (and thus is in the opposite order to walk_stack). Usually used with StackSummary.extract. """ while tb is not None: > yield tb.tb_frame, tb.tb_lineno E AttributeError: 'str' object has no attribute 'tb_frame' /usr/lib64/python3.9/traceback.py:312: AttributeError _____________ TestFlakyPlugin.test_flaky_plugin_identifies_failure _____________ self = def test_flaky_plugin_identifies_failure(self): > for name, test in self._test_dataset: E ValueError: too many values to unpack (expected 2) test/test_flaky_plugin.py:45: ValueError ________________ TestMultiprocessStringIO.test_write_then_read _________________ self = def test_write_then_read(self): for name, value in self._test_values.items(): with self.subTest(name): for string_io in self._string_ios: for item in value[0]: string_io.write(item) > self.assertEqual(string_io.getvalue(), value[1]) E AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' E - Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ E ? ----------- E + Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ test/test_multiprocess_string_io.py:31: AssertionError ______________ TestMultiprocessStringIO.test_writelines_then_read ______________ self = def test_writelines_then_read(self): for name, value in self._test_values.items(): with self.subTest(name): for string_io in self._string_ios: string_io.writelines(value[0]) > self.assertEqual(string_io.getvalue(), value[1]) E AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' E - Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ E ? ----------- E + Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ test/test_multiprocess_string_io.py:38: AssertionError _____________________________ test_something_flaky _____________________________ dummy_list = [0] def test_something_flaky(dummy_list=[]): # pylint:disable=dangerous-default-value dummy_list.append(0) > assert len(dummy_list) > 1 E assert 1 > 1 E + where 1 = len([0]) test/test_pytest/test_pytest_options_example.py:11: AssertionError ================================== XFAILURES =================================== ________________ test_something_good_with_failing_setup_fixture ________________ @pytest.fixture(scope='function') def failing_setup_fixture(): > assert False E assert False test/test_pytest/test_pytest_example.py:21: AssertionError ________________ test_something_good_with_failing_setup_fixture ________________ @pytest.fixture(scope='function') def failing_setup_fixture(): > assert False E assert False test/test_pytest/test_pytest_example.py:21: AssertionError ___________________ TestExample.test_non_flaky_failing_thing ___________________ self = @pytest.mark.xfail def test_non_flaky_failing_thing(self): """Flaky will also not interact with this test""" > assert self == 1 E assert == 1 test/test_pytest/test_pytest_example.py:40: AssertionError =============================== warnings summary =============================== test/test_flaky_plugin.py:0 /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_flaky_plugin.py:0: PytestCollectionWarning: cannot collect test class 'TestCaseDataset' because it has a __new__ constructor (from: test/test_flaky_plugin.py) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ===Flaky Test Report=== Flaky report texttest_something_flaky failed (1 runs remaining out of 2). assert 1 > 1 + where 1 = len([0]) [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_something_flaky passed 1 out of the required 1 times. Success! test_something_good_with_failing_setup_fixture failed (1 runs remaining out of 2). assert False [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_something_good_with_failing_setup_fixture failed; it passed 0 out of the required 1 times. assert False [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3). assert 0 >= 1 + where 0 = TestExample._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times. test_flaky_thing_that_fails_then_succeeds passed 2 out of the required 2 times. Success! test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times. test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 2 out of the required 2 times. Success! test_flaky_thing_that_always_passes passed 1 out of the required 2 times. Running test again until it passes 2 times. test_flaky_thing_that_always_passes passed 2 out of the required 2 times. Success! test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestExampleFlakyTests._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestExampleFlakyTestCase._threshold [, , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestMarkedClass._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_requiring_my_fixture failed (1 runs remaining out of 2). assert 1 > 1 + where 1 = len([0]) [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_requiring_my_fixture passed 1 out of the required 1 times. Success! test_something_flaky failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestExampleRerunFilter._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_something_flaky passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3). assert -1 >= 1 + where -1 = TestExample._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3). assert 0 >= 1 + where 0 = TestExample._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3). assert -1 >= 1 + where -1 = TestExampleFlakyTests._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3). assert 0 >= 1 + where 0 = TestExampleFlakyTests._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! ===End Flaky Test Report=== =========================== short test summary info ============================ SKIPPED [1] test/test_pytest/test_pytest_example.py:66: This really fails! Remove skipif to see the test failure. SKIPPED [1] ../../../../../usr/lib/python3.9/site-packages/_pytest/unittest.py:357: This test always fails XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture XFAIL test/test_pytest/test_pytest_example.py::TestExample::test_non_flaky_failing_thing FAILED test/test_flaky_plugin.py::TestFlakyPlugin::test_flaky_plugin_handles_non_ascii_byte_string_in_exception FAILED test/test_flaky_plugin.py::TestFlakyPlugin::test_flaky_plugin_identifies_failure FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_write_then_read FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_writelines_then_read FAILED test/test_pytest/test_pytest_options_example.py::test_something_flaky ======== 5 failed, 68 passed, 2 skipped, 3 xfailed, 1 warning in 0.44s ========= ```
mtelka commented 6 months ago

FYI, I tested flaky 3.8.0 on OpenIndiana and all tests passed here. Tested with pytest 8.1.1.

kloczek commented 6 months ago

One se c I'll try to retest thah ton 8.1.1 😋

kloczek commented 6 months ago

Nope it still fails

Here is pytest output: ```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages + /usr/bin/pytest -ra -m 'not network' ==================================================================================== test session starts ==================================================================================== platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0 rootdir: /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0 plugins: flaky-3.7.0 collected 77 items test/test_flaky_decorator.py ... [ 3%] test/test_flaky_plugin.py FF. [ 7%] test/test_multiprocess_string_io.py FF [ 10%] test/test_pytest/pytest_generate_example/test_pytest_generate_example.py .. [ 12%] test/test_pytest/test_flaky_pytest_plugin.py ................................................ [ 75%] test/test_pytest/test_pytest_example.py .xx.x...s......s [ 94%] test/test_pytest/test_pytest_options_example.py F.. [ 98%] test/test_pytest/test_pytester_plugin.py . [100%] ========================================================================================= FAILURES ========================================================================================== _______________________________________________________ TestFlakyPlugin.test_flaky_plugin_handles_non_ascii_byte_string_in_exception ________________________________________________________ self = def test_flaky_plugin_handles_non_ascii_byte_string_in_exception(self): mock_method_name = 'my_method' mock_exception = 'ńőń ȁŝćȉȉ ŝƭȕƒƒ'.encode('utf-16') mock_message = 'information about retries' # pylint:disable=protected-access > self._flaky_plugin._log_test_failure( mock_method_name, (ValueError.__name__, mock_exception, ''), mock_message, ) test/test_flaky_plugin.py:38: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ ../../BUILDROOT/python-flaky-3.8.0-2.fc36.x86_64/usr/lib/python3.9/site-packages/flaky/_flaky_plugin.py:38: in _log_test_failure formatted_exception_info = ''.join(format_exception(*err)).replace('\n', '\n\t').rstrip() /usr/lib64/python3.9/traceback.py:120: in format_exception return list(TracebackException( /usr/lib/python3.9/site-packages/exceptiongroup/_formatting.py:96: in __init__ self.stack = traceback.StackSummary.extract( /usr/lib64/python3.9/traceback.py:347: in extract for f, lineno in frame_gen: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ tb = '' def walk_tb(tb): """Walk a traceback yielding the frame and line number for each frame. This will follow tb.tb_next (and thus is in the opposite order to walk_stack). Usually used with StackSummary.extract. """ while tb is not None: > yield tb.tb_frame, tb.tb_lineno E AttributeError: 'str' object has no attribute 'tb_frame' /usr/lib64/python3.9/traceback.py:312: AttributeError ___________________________________________________________________ TestFlakyPlugin.test_flaky_plugin_identifies_failure ____________________________________________________________________ self = def test_flaky_plugin_identifies_failure(self): > for name, test in self._test_dataset: E ValueError: too many values to unpack (expected 2) test/test_flaky_plugin.py:45: ValueError _______________________________________________________________________ TestMultiprocessStringIO.test_write_then_read _______________________________________________________________________ self = def test_write_then_read(self): for name, value in self._test_values.items(): with self.subTest(name): for string_io in self._string_ios: for item in value[0]: string_io.write(item) > self.assertEqual(string_io.getvalue(), value[1]) E AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' E - Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ E ? ----------- E + Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ test/test_multiprocess_string_io.py:31: AssertionError ____________________________________________________________________ TestMultiprocessStringIO.test_writelines_then_read _____________________________________________________________________ self = def test_writelines_then_read(self): for name, value in self._test_values.items(): with self.subTest(name): for string_io in self._string_ios: string_io.writelines(value[0]) > self.assertEqual(string_io.getvalue(), value[1]) E AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' E - Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ E ? ----------- E + Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ test/test_multiprocess_string_io.py:38: AssertionError ___________________________________________________________________________________ test_something_flaky ____________________________________________________________________________________ dummy_list = [0] def test_something_flaky(dummy_list=[]): # pylint:disable=dangerous-default-value dummy_list.append(0) > assert len(dummy_list) > 1 E assert 1 > 1 E + where 1 = len([0]) test/test_pytest/test_pytest_options_example.py:11: AssertionError ========================================================================================= XFAILURES ========================================================================================= ______________________________________________________________________ test_something_good_with_failing_setup_fixture _______________________________________________________________________ @pytest.fixture(scope='function') def failing_setup_fixture(): > assert False E assert False test/test_pytest/test_pytest_example.py:21: AssertionError ______________________________________________________________________ test_something_good_with_failing_setup_fixture _______________________________________________________________________ @pytest.fixture(scope='function') def failing_setup_fixture(): > assert False E assert False test/test_pytest/test_pytest_example.py:21: AssertionError _________________________________________________________________________ TestExample.test_non_flaky_failing_thing __________________________________________________________________________ self = @pytest.mark.xfail def test_non_flaky_failing_thing(self): """Flaky will also not interact with this test""" > assert self == 1 E assert == 1 test/test_pytest/test_pytest_example.py:40: AssertionError ===================================================================================== warnings summary ====================================================================================== test/test_flaky_plugin.py:0 /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0/test/test_flaky_plugin.py:0: PytestCollectionWarning: cannot collect test class 'TestCaseDataset' because it has a __new__ constructor (from: test/test_flaky_plugin.py) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ===Flaky Test Report=== Flaky report texttest_something_flaky failed (1 runs remaining out of 2). assert 1 > 1 + where 1 = len([0]) [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_something_flaky passed 1 out of the required 1 times. Success! test_something_good_with_failing_setup_fixture failed (1 runs remaining out of 2). assert False [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_something_good_with_failing_setup_fixture failed; it passed 0 out of the required 1 times. assert False [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3). assert 0 >= 1 + where 0 = TestExample._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times. test_flaky_thing_that_fails_then_succeeds passed 2 out of the required 2 times. Success! test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times. test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 2 out of the required 2 times. Success! test_flaky_thing_that_always_passes passed 1 out of the required 2 times. Running test again until it passes 2 times. test_flaky_thing_that_always_passes passed 2 out of the required 2 times. Success! test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestExampleFlakyTests._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestExampleFlakyTestCase._threshold [, , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestMarkedClass._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_requiring_my_fixture failed (1 runs remaining out of 2). assert 1 > 1 + where 1 = len([0]) [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_requiring_my_fixture passed 1 out of the required 1 times. Success! test_something_flaky failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestExampleRerunFilter._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_something_flaky passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3). assert -1 >= 1 + where -1 = TestExample._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3). assert 0 >= 1 + where 0 = TestExample._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3). assert -1 >= 1 + where -1 = TestExampleFlakyTests._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3). assert 0 >= 1 + where 0 = TestExampleFlakyTests._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! ===End Flaky Test Report=== ================================================================================== short test summary info ================================================================================== SKIPPED [1] test/test_pytest/test_pytest_example.py:66: This really fails! Remove skipif to see the test failure. SKIPPED [1] ../../../../../usr/lib/python3.9/site-packages/_pytest/unittest.py:357: This test always fails XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture XFAIL test/test_pytest/test_pytest_example.py::TestExample::test_non_flaky_failing_thing FAILED test/test_flaky_plugin.py::TestFlakyPlugin::test_flaky_plugin_handles_non_ascii_byte_string_in_exception - AttributeError: 'str' object has no attribute 'tb_frame' FAILED test/test_flaky_plugin.py::TestFlakyPlugin::test_flaky_plugin_identifies_failure - ValueError: too many values to unpack (expected 2) FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_write_then_read - AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_writelines_then_read - AssertionError: 'Plain HelloPlain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' != 'Plain Hellońőń ȁŝćȉȉ ŝƭȕƒƒ' FAILED test/test_pytest/test_pytest_options_example.py::test_something_flaky - assert 1 > 1 =============================================================== 5 failed, 68 passed, 2 skipped, 3 xfailed, 1 warning in 0.47s ===============================================================```
List of installed modules in build env: ```console Package Version ------------------ ----------- build 1.1.1 distro 1.9.0 exceptiongroup 1.1.3 importlib_metadata 7.0.1 iniconfig 2.0.0 installer 0.7.0 packaging 24.0 pluggy 1.4.0 pyproject_hooks 1.0.0 pytest 8.1.1 python-dateutil 2.9.0.post0 setuptools 69.1.1 tokenize_rt 5.2.0 tomli 2.0.1 wheel 0.42.0 zipp 3.17.0 ```
kloczek commented 6 months ago

Just spotted odd detail in my output:

+ /usr/bin/pytest -ra -m 'not network'
==================================================================================== test session starts ====================================================================================
platform linux -- Python 3.9.18, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/tkloczko/rpmbuild/BUILD/flaky-3.8.0           <<< so here is root directory with version 3.8.0 
plugins: flaky-3.7.0                                         <<< but here pytest reports version 3.7.0
kloczek commented 6 months ago

and indeed ..

[tkloczko@pers-jacek flaky-3.8.0]$ grep -r 3.7.0
HISTORY.rst:3.7.0 (2020-07-07)
setup.py:        version='3.7.0',
kloczek commented 6 months ago

https://github.com/box/flaky/blob/v3.8.0/setup.py#L52-L54

mtelka commented 6 months ago

and indeed ..

[tkloczko@pers-jacek flaky-3.8.0]$ grep -r 3.7.0
HISTORY.rst:3.7.0 (2020-07-07)
setup.py:        version='3.7.0',

This is #202.

mtelka commented 6 months ago

I believe the problem reported by OP is already fixed in flaky 3.8.0 via #201.