search

simplistix / sybil

Automated testing for the examples in your documentation.
https://sybil.readthedocs.io/en/latest/
Other
74 stars 14 forks source link

pytest: `IndexError: list index out of range` when filtering traceback at `tb_entry = tb[1]` #118

Closed nstarman closed 4 months ago

nstarman commented 4 months ago

In some CI tests pluggy is raising an error about test teardown. I'm not entirely convinced this is sybil causing this, but in the following log it's the only pytest integration I'm seeing.

5_Test package.txt from https://github.com/GalacticDynamics/galax/actions/runs/9546135066/job/26434625048?pr=355

cjw296 commented 4 months ago

Can you reproduce this locally running the same stuff as you have there? I do see Sybil in the traceback...

cjw296 commented 4 months ago

Put differently, if you can give me a list of instructions to reproduce, I'd like to take a look, eg:

git clone ???
cd ???
python3.12 -m venv ???
pip? install? -r ???
pytest ???

I believe there may well be a bug in Sybil lurking here :-)

cjw296 commented 4 months ago

For reference, this is the error:

Run python -m pytest src docs tests/unit tests/functional -ra --cov --cov-report=xml --cov-report=term --durations=20 --arraydiff -m"not slow"
============================= test session starts ==============================
platform linux -- Python 3.12.4, pytest-7.4.4, pluggy-1.5.0
rootdir: /home/runner/work/galax/galax
configfile: pyproject.toml
plugins: jaxtyping-0.2.30, arraydiff-0.6.1, typeguard-2.13.3, hypothesis-6.103.2, cov-5.0.0
collected 2012 items / 1 deselected / 2011 selected

src/galax/coordinates/_compat.py 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 156, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/skipping.py", line 266, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/runner.py", line 368, in pytest_runtest_makereport
INTERNALERROR>     return TestReport.from_item_and_call(item, call)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/reports.py", line 362, in from_item_and_call
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>                ^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/sybil/integration/pytest.py", line 117, in repr_failure
INTERNALERROR>     return super().repr_failure(excinfo, style)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/nodes.py", line 506, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/nodes.py", line 486, in _repr_failure_py
INTERNALERROR>     return excinfo.getrepr(
INTERNALERROR>            ^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/_code/code.py", line 701, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/_code/code.py", line 989, in repr_excinfo
INTERNALERROR>     reprtraceback = self.repr_traceback(excinfo_)
INTERNALERROR>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/_code/code.py", line 893, in repr_traceback
INTERNALERROR>     traceback = self.tbfilter(excinfo)
INTERNALERROR>                 ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/sybil/integration/pytest.py", line 97, in _traceback_filter
INTERNALERROR>     tb_entry = tb[1]
INTERNALERROR>                ~~^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/_code/code.py", line 395, in __getitem__
INTERNALERROR>     return super().__getitem__(key)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> IndexError: list index out of range
INTERNALERROR> 
INTERNALERROR> During handling of the above exception, another exception occurred:
INTERNALERROR> 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/runner.py", line 133, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/_pytest/runner.py", line 224, in call_and_report
INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 160, in _multicall
INTERNALERROR>     _warn_teardown_exception(hook_name, teardown[1], e)
INTERNALERROR>   File "/opt/hostedtoolcache/Python/3.12.4/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 50, in _warn_teardown_exception
INTERNALERROR>     warnings.warn(PluggyTeardownRaisedWarning(msg), stacklevel=5)
INTERNALERROR> pluggy.PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
INTERNALERROR> Plugin: skipping, Hook: pytest_runtest_makereport
INTERNALERROR> IndexError: list index out of range
INTERNALERROR> For more information see
https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning

...but without instructions to reproduce, or confirmation that the problem is still happening, I'll close this as "not planned" in a few days time

nstarman commented 4 months ago

I can confirm it's still happening. I'm having trouble finding a MWE, because the error is only happening when I do a full-dependency CI run, not under normal circumstances. I'll have some time in a week or so.

cjw296 commented 4 months ago

Okay, closing for now, will re-open once a reproducer is provided.