search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
12.02k stars 2.67k forks source link

INTERNALERROR> IndexError: list index out of range in hypothesis and pytest-bdd #10903

Closed bluetech closed 1 year ago

bluetech commented 1 year ago

Discussed in https://github.com/pytest-dev/pytest/discussions/10895

Originally posted by **hroncok** April 11, 2023 Hello pytest folks. When [testing the impact of the pytest upgrade form 7.2.x to 7.3.0 in Fedora](https://src.fedoraproject.org/rpms/pytest/pull-request/42#comment-136827), I found two interesting failures in https://copr.fedorainfracloud.org/coprs/churchyard/pytest-7.3.0/package/python-hypothesis/ 6.62.1 ``` platform linux -- Python 3.11.3, pytest-7.3.0, pluggy-1.0.0 rootdir: /tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_suppress_health_check_function_scoped_fixture0 plugins: hypothesis-6.62.1, xdist-3.2.0 collected 3 items test_suppress_health_check_function_scoped_fixture.py INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 269, in wrap_session INTERNALERROR> session.exitstatus = doit(config, session) or 0 INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 323, in _main INTERNALERROR> config.hook.pytest_runtestloop(session=session) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__ INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall INTERNALERROR> return outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex[1].with_traceback(ex[2]) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 348, in pytest_runtestloop INTERNALERROR> item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__ INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall INTERNALERROR> return outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex[1].with_traceback(ex[2]) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol INTERNALERROR> runtestprotocol(item, nextitem=nextitem) INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol INTERNALERROR> reports.append(call_and_report(item, "call", log)) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 224, in call_and_report INTERNALERROR> report: TestReport = hook.pytest_runtest_makereport(item=item, call=call) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__ INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 55, in _multicall INTERNALERROR> gen.send(outcome) INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/skipping.py", line 266, in pytest_runtest_makereport INTERNALERROR> rep = outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex[1].with_traceback(ex[2]) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 368, in pytest_runtest_makereport INTERNALERROR> return TestReport.from_item_and_call(item, call) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/reports.py", line 363, in from_item_and_call INTERNALERROR> longrepr = item.repr_failure(excinfo) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 1833, in repr_failure INTERNALERROR> return self._repr_failure_py(excinfo, style=style) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/nodes.py", line 484, in _repr_failure_py INTERNALERROR> return excinfo.getrepr( INTERNALERROR> ^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 671, in getrepr INTERNALERROR> return fmt.repr_excinfo(self) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 987, in repr_excinfo INTERNALERROR> return ExceptionChainRepr(repr_chain) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 1039, in __init__ INTERNALERROR> reprtraceback=chain[-1][0], INTERNALERROR> ~~~~~^^^^ INTERNALERROR> IndexError: list index out of range ``` And with https://copr.fedorainfracloud.org/coprs/churchyard/pytest-7.3.0/package/python-pytest-bdd/ 6.1.1 ``` platform linux -- Python 3.11.3, pytest-7.3.0, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /tmp/pytest-of-mockbuild/pytest-0/popen-gw1/test_step_trace2 configfile: tox.ini plugins: bdd-6.1.1, xdist-3.2.0 collecting ... collected 4 items / 3 deselected / 1 selected test_step_trace.py::test_when_not_found <- ../../../../../builddir/build/BUILDROOT/python-pytest-bdd-6.1.1-7.fc39.x86_64/usr/lib/python3.11/site-packages/pytest_bdd/scenario.py INTERNALERROR> Traceback (most recent call last): INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 269, in wrap_session INTERNALERROR> session.exitstatus = doit(config, session) or 0 INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 323, in _main INTERNALERROR> config.hook.pytest_runtestloop(session=session) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__ INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 327, in traced_hookexec INTERNALERROR> return outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex[1].with_traceback(ex[2]) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 33, in from_call INTERNALERROR> result = func() INTERNALERROR> ^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 324, in INTERNALERROR> lambda: oldcall(hook_name, hook_impls, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall INTERNALERROR> return outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex[1].with_traceback(ex[2]) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/main.py", line 348, in pytest_runtestloop INTERNALERROR> item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__ INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 327, in traced_hookexec INTERNALERROR> return outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex[1].with_traceback(ex[2]) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 33, in from_call INTERNALERROR> result = func() INTERNALERROR> ^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 324, in INTERNALERROR> lambda: oldcall(hook_name, hook_impls, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall INTERNALERROR> return outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex[1].with_traceback(ex[2]) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol INTERNALERROR> runtestprotocol(item, nextitem=nextitem) INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol INTERNALERROR> reports.append(call_and_report(item, "call", log)) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 224, in call_and_report INTERNALERROR> report: TestReport = hook.pytest_runtest_makereport(item=item, call=call) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in __call__ INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 327, in traced_hookexec INTERNALERROR> return outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex[1].with_traceback(ex[2]) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 33, in from_call INTERNALERROR> result = func() INTERNALERROR> ^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_manager.py", line 324, in INTERNALERROR> lambda: oldcall(hook_name, hook_impls, kwargs, firstresult) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 55, in _multicall INTERNALERROR> gen.send(outcome) INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/skipping.py", line 266, in pytest_runtest_makereport INTERNALERROR> rep = outcome.get_result() INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result INTERNALERROR> raise ex[1].with_traceback(ex[2]) INTERNALERROR> File "/usr/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall INTERNALERROR> res = hook_impl.function(*args) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/runner.py", line 368, in pytest_runtest_makereport INTERNALERROR> return TestReport.from_item_and_call(item, call) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/reports.py", line 363, in from_item_and_call INTERNALERROR> longrepr = item.repr_failure(excinfo) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/python.py", line 1833, in repr_failure INTERNALERROR> return self._repr_failure_py(excinfo, style=style) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/nodes.py", line 484, in _repr_failure_py INTERNALERROR> return excinfo.getrepr( INTERNALERROR> ^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 671, in getrepr INTERNALERROR> return fmt.repr_excinfo(self) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 987, in repr_excinfo INTERNALERROR> return ExceptionChainRepr(repr_chain) INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ INTERNALERROR> File "/usr/lib/python3.11/site-packages/_pytest/_code/code.py", line 1039, in __init__ INTERNALERROR> reprtraceback=chain[-1][0], INTERNALERROR> ~~~~~^^^^ INTERNALERROR> IndexError: list index out of range ``` I will try to retest with the latest versions of hypothesis, but pytest-bdd is up to date. I am puzzleed on where to report this. Is this a bug in pytest? Or in the tests?
bluetech commented 1 year ago

As discussed in the linked discussion, this is a regression from PR #10772.

This issue can be reduced to the following:

def test():
    __tracebackhide__ = True
    1 / 0

this crashes with INTERNALERROR> IndexError: list index out of range as reported above.

The code handling exception formatting is definitely broken in how it handles __cause__/__context__ exceptions. The "main" exception goes through the Node _prunetraceback code, which then passes tbfilter=False because it's "pruned already", but that neglects the chained exceptions entirely.

The way #10772 handled it is to use getcrashentry() being None as a proxy for omitting the exception. I don't believe this is the correct way to do it, instead we need to have the main and chained exceptions go through the same pruning code so that everything is consistent.

Further, we need to decide what should happen in the reproduction above, where all frames in the exception are hidden -- do we show the top frame anyway (status quo), show some special message, or just show nothing?

For now, I am going to send a PR reverting 431ec6d34ef99d80f90b330876ed6231144a3ce7 because whatever the fix is, it's not going to be very simple and should go into the next feature release.

bluetech commented 1 year ago

Also relevant: accd962c9f88dbd5b2b0eef6efe7bf6fe5444b29 issue #1604