Fix incompatibility with pytest == 8.1.0

[jonathan-eq]

Fixes #198

Fixes incompatibility with pytest == 8.1.0 by internalizing call_runtest_hook function.

[CLAassistant]

All committers have signed the CLA.

[Jeff-Meadows]

The idea of the change here looks good but it has some syntax errors and codestyle errors. Could you clean those up?

[jnewb1]

The idea of the change here looks good but it has some syntax errors and codestyle errors. Could you clean those up?

the suggestions above should fix the errors

[mgorny]

FYI, 8.1.0 has been yanked and the removals have been reverted for the time being, FWIU at least until they're better documented.

[kloczek]

Tested this PR on top of current master and pytest fails with

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.7.0-15.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.7.0-15.fc36.x86_64/usr/lib/python3.9/site-packages + /usr/bin/pytest -ra -m 'not network' ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.0, pluggy-1.4.0 rootdir: /home/tkloczko/rpmbuild/BUILD/flaky-3.7.0 plugins: flaky-3.7.0 collected 74 items / 1 error ==================================== ERRORS ==================================== __________________ ERROR collecting test/test_flaky_plugin.py __________________ /usr/lib/python3.9/site-packages/_pytest/python.py:524: in importtestmodule mod = import_path( /usr/lib/python3.9/site-packages/_pytest/pathlib.py:580: in import_path importlib.import_module(module_name) /usr/lib64/python3.9/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) :1030: in _gcd_import ??? :1007: in _find_and_load ??? :986: in _find_and_load_unlocked ??? :680: in _load_unlocked ??? /usr/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:169: in exec_module source_stat, co = _rewrite_test(fn, self.config) /usr/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:351: in _rewrite_test tree = ast.parse(source, filename=strfn) /usr/lib64/python3.9/ast.py:50: in parse return compile(source, filename, mode, flags, E File "/home/tkloczko/rpmbuild/BUILD/flaky-3.7.0/test/test_flaky_plugin.py", line 13 E "default_not_started": TestCaseDataset(2, 1, 0, 0, False), E ^ E SyntaxError: invalid syntax =========================== short test summary info ============================ ERROR test/test_flaky_plugin.py !!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!! =============================== 1 error in 0.30s =============================== ```

[kloczek]

After add test/test_flaky_plugin.py to --ignore list pytest passes units scan but it fails in 6 units

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.7.0-15.fc36.x86_64/usr/lib64/python3.9/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-flaky-3.7.0-15.fc36.x86_64/usr/lib/python3.9/site-packages + /usr/bin/pytest -ra -m 'not network' --ignore test/test_flaky_plugin.py ============================= test session starts ============================== platform linux -- Python 3.9.18, pytest-8.1.0, pluggy-1.4.0 rootdir: /home/tkloczko/rpmbuild/BUILD/flaky-3.7.0 plugins: flaky-3.7.0 collected 74 items test/test_flaky_decorator.py ... [ 4%] test/test_multiprocess_string_io.py FF [ 6%] test/test_pytest/pytest_generate_example/test_pytest_generate_example.py [ 8%] .. [ 9%] test/test_pytest/test_flaky_pytest_plugin.py .....................FFF... [ 45%] ..................... [ 74%] test/test_pytest/test_pytest_example.py .xx.x...s......s [ 94%] test/test_pytest/test_pytest_options_example.py F.. [ 98%] test/test_pytest/test_pytester_plugin.py . [100%] =================================== FAILURES =================================== ________________ TestMultiprocessStringIO.test_write_then_read _________________ self = def test_write_then_read(self): > for name in _test_values: E NameError: name '_test_values' is not defined test/test_multiprocess_string_io.py:26: NameError ______________ TestMultiprocessStringIO.test_writelines_then_read ______________ self = def test_writelines_then_read(self): > for name in _test_values: E NameError: name '_test_values' is not defined test/test_multiprocess_string_io.py:34: NameError __________ test_flaky_plugin_raises_errors_in_fixture_setup[instance] __________ flaky_test = flaky_plugin = string_io = <_io.StringIO object at 0x7f3fcebce4c0> mock_io = <_io.StringIO object at 0x7f3fcebce670> def test_flaky_plugin_raises_errors_in_fixture_setup( flaky_test, flaky_plugin, string_io, mock_io, ): """ Test for Issue #57 - fixtures which raise an error should show up as test errors. This test ensures that exceptions occurring when running a test fixture are copied into the call info's excinfo field. """ def error_raising_setup_function(item): assert item is flaky_test item.ran_setup = True return 5 / 0 flaky()(flaky_test) flaky_test.ihook = Mock() flaky_test.ihook.pytest_runtest_setup = error_raising_setup_function flaky_plugin._call_infos[flaky_test] = {} # pylint:disable=protected-access > call_info = runner.call_runtest_hook(flaky_test, 'setup') E AttributeError: module '_pytest.runner' has no attribute 'call_runtest_hook' test/test_pytest/test_flaky_pytest_plugin.py:242: AttributeError ___________ test_flaky_plugin_raises_errors_in_fixture_setup[module] ___________ flaky_test = flaky_plugin = string_io = <_io.StringIO object at 0x7f3fcf7c4430> mock_io = <_io.StringIO object at 0x7f3fcf7c4550> def test_flaky_plugin_raises_errors_in_fixture_setup( flaky_test, flaky_plugin, string_io, mock_io, ): """ Test for Issue #57 - fixtures which raise an error should show up as test errors. This test ensures that exceptions occurring when running a test fixture are copied into the call info's excinfo field. """ def error_raising_setup_function(item): assert item is flaky_test item.ran_setup = True return 5 / 0 flaky()(flaky_test) flaky_test.ihook = Mock() flaky_test.ihook.pytest_runtest_setup = error_raising_setup_function flaky_plugin._call_infos[flaky_test] = {} # pylint:disable=protected-access > call_info = runner.call_runtest_hook(flaky_test, 'setup') E AttributeError: module '_pytest.runner' has no attribute 'call_runtest_hook' test/test_pytest/test_flaky_pytest_plugin.py:242: AttributeError ___________ test_flaky_plugin_raises_errors_in_fixture_setup[parent] ___________ flaky_test = flaky_plugin = string_io = <_io.StringIO object at 0x7f3fcf7c2ee0> mock_io = <_io.StringIO object at 0x7f3fcec9faf0> def test_flaky_plugin_raises_errors_in_fixture_setup( flaky_test, flaky_plugin, string_io, mock_io, ): """ Test for Issue #57 - fixtures which raise an error should show up as test errors. This test ensures that exceptions occurring when running a test fixture are copied into the call info's excinfo field. """ def error_raising_setup_function(item): assert item is flaky_test item.ran_setup = True return 5 / 0 flaky()(flaky_test) flaky_test.ihook = Mock() flaky_test.ihook.pytest_runtest_setup = error_raising_setup_function flaky_plugin._call_infos[flaky_test] = {} # pylint:disable=protected-access > call_info = runner.call_runtest_hook(flaky_test, 'setup') E AttributeError: module '_pytest.runner' has no attribute 'call_runtest_hook' test/test_pytest/test_flaky_pytest_plugin.py:242: AttributeError _____________________________ test_something_flaky _____________________________ dummy_list = [0] def test_something_flaky(dummy_list=[]): # pylint:disable=dangerous-default-value dummy_list.append(0) > assert len(dummy_list) > 1 E assert 1 > 1 E + where 1 = len([0]) test/test_pytest/test_pytest_options_example.py:11: AssertionError ================================== XFAILURES =================================== ________________ test_something_good_with_failing_setup_fixture ________________ @pytest.fixture(scope='function') def failing_setup_fixture(): > assert False E assert False test/test_pytest/test_pytest_example.py:21: AssertionError ________________ test_something_good_with_failing_setup_fixture ________________ @pytest.fixture(scope='function') def failing_setup_fixture(): > assert False E assert False test/test_pytest/test_pytest_example.py:21: AssertionError ___________________ TestExample.test_non_flaky_failing_thing ___________________ self = @pytest.mark.xfail def test_non_flaky_failing_thing(self): """Flaky will also not interact with this test""" > assert self == 1 E assert == 1 test/test_pytest/test_pytest_example.py:40: AssertionError =============================== warnings summary =============================== test/test_flaky_decorator.py::TestFlakyDecorator::test_flaky_adds_flaky_attributes_to_test_method /usr/lib64/python3.9/unittest/case.py:1140: DeprecationWarning: assertDictContainsSubset is deprecated warnings.warn('assertDictContainsSubset is deprecated', -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ===Flaky Test Report=== Flaky report texttest_something_flaky failed (1 runs remaining out of 2). assert 1 > 1 + where 1 = len([0]) [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_something_flaky passed 1 out of the required 1 times. Success! test_something_good_with_failing_setup_fixture failed (1 runs remaining out of 2). assert False [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_something_good_with_failing_setup_fixture failed; it passed 0 out of the required 1 times. assert False [, , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3). assert 0 >= 1 + where 0 = TestExample._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times. test_flaky_thing_that_fails_then_succeeds passed 2 out of the required 2 times. Success! test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 1 out of the required 2 times. Running test again until it passes 2 times. test_flaky_thing_that_succeeds_then_fails_then_succeeds passed 2 out of the required 2 times. Success! test_flaky_thing_that_always_passes passed 1 out of the required 2 times. Running test again until it passes 2 times. test_flaky_thing_that_always_passes passed 2 out of the required 2 times. Success! test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestExampleFlakyTests._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestExampleFlakyTestCase._threshold [, , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestMarkedClass._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_requiring_my_fixture failed (1 runs remaining out of 2). assert 1 > 1 + where 1 = len([0]) [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_requiring_my_fixture passed 1 out of the required 1 times. Success! test_something_flaky failed (1 runs remaining out of 2). assert 0 >= 1 + where 0 = TestExampleRerunFilter._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_something_flaky passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3). assert -1 >= 1 + where -1 = TestExample._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3). assert 0 >= 1 + where 0 = TestExample._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! test_flaky_thing_that_fails_then_succeeds failed (2 runs remaining out of 3). assert -1 >= 1 + where -1 = TestExampleFlakyTests._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 3). assert 0 >= 1 + where 0 = TestExampleFlakyTests._threshold [, , , , , , , , , , , , , , , , , , , , , , , , , , , ] test_flaky_thing_that_fails_then_succeeds passed 1 out of the required 1 times. Success! ===End Flaky Test Report=== =========================== short test summary info ============================ SKIPPED [1] test/test_pytest/test_pytest_example.py:66: This really fails! Remove skipif to see the test failure. SKIPPED [1] ../../../../../usr/lib/python3.9/site-packages/_pytest/unittest.py:357: This test always fails XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture XFAIL test/test_pytest/test_pytest_example.py::test_something_good_with_failing_setup_fixture XFAIL test/test_pytest/test_pytest_example.py::TestExample::test_non_flaky_failing_thing FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_write_then_read FAILED test/test_multiprocess_string_io.py::TestMultiprocessStringIO::test_writelines_then_read FAILED test/test_pytest/test_flaky_pytest_plugin.py::test_flaky_plugin_raises_errors_in_fixture_setup[instance] FAILED test/test_pytest/test_flaky_pytest_plugin.py::test_flaky_plugin_raises_errors_in_fixture_setup[module] FAILED test/test_pytest/test_flaky_pytest_plugin.py::test_flaky_plugin_raises_errors_in_fixture_setup[parent] FAILED test/test_pytest/test_pytest_options_example.py::test_something_flaky ======== 6 failed, 64 passed, 2 skipped, 3 xfailed, 1 warning in 0.42s ========= ```

[Jeff-Meadows]

@jonathan-eq and others, thanks for the help. I needed to fix some problems with flaky on Python 3.12 in order to get CI passing, so I did that on another branch in #201, but I was able to use the commit from this PR.

I'll try to create a new release later today.