search

CodSpeedHQ / pytest-codspeed

Pytest plugin to create CodSpeed benchmarks
https://codspeed.io
MIT License
58 stars 7 forks source link

`pytest-codspeed` crashing since `pytest` 8.1.1 #27

Closed davidhewitt closed 7 months ago

davidhewitt commented 7 months ago

PyO3 benches jobs in CI have recently started failing, somewhere in the pytest stage.

It looks like pytest 8.1.1 has changed something incompatibly with pytest-codspeed. I was able to fix the problem in PyO3's CI by pinning back to pytest < 8.1.

Here's a dump of the error:

nox > pytest --codspeed
  INTERNALERROR> Traceback (most recent call last):
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/_pytest/main.py", line 283, in wrap_session
  INTERNALERROR>     config.hook.pytest_sessionstart(session=session)
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__
  INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec
  INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pluggy/_callers.py", line 138, in _multicall
  INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
  INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
  INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/_pytest/logging.py", line 785, in pytest_sessionstart
  INTERNALERROR>     return (yield)
  INTERNALERROR>             ^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
  INTERNALERROR>     res = hook_impl.function(*args)
  INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/_pytest/fixtures.py", line 118, in pytest_sessionstart
  INTERNALERROR>     session._fixturemanager = FixtureManager(session)
  INTERNALERROR>                               ^^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/_pytest/fixtures.py", line 1438, in __init__
  INTERNALERROR>     session.config.pluginmanager.register(self, "funcmanage")
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/_pytest/config/__init__.py", line 501, in register
  INTERNALERROR>     self.hook.pytest_plugin_registered.call_historic(
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pluggy/_hooks.py", line 523, in call_historic
  INTERNALERROR>     res = self._hookexec(self.name, self._hookimpls.copy(), kwargs, False)
  INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec
  INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pluggy/_callers.py", line 138, in _multicall
  INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
  INTERNALERROR>     res = hook_impl.function(*args)
  INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/pytest_codspeed/plugin.py", line 97, in pytest_plugin_registered
  INTERNALERROR>     codspeed_benchmark_fixtures = plugin.getfixturedefs(
  INTERNALERROR>                                   ^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/_pytest/fixtures.py", line 1765, in getfixturedefs
  INTERNALERROR>     return tuple(self._matchfactories(fixturedefs, node))
  INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  INTERNALERROR>   File "/home/runner/work/pyo3/pyo3/.nox/codspeed/lib/python3.12/site-packages/_pytest/fixtures.py", line 1770, in _matchfactories
  INTERNALERROR>     parentnodeids = {n.nodeid for n in node.iter_parents()}
  INTERNALERROR>                                        ^^^^^^^^^^^^^^^^^
  INTERNALERROR> AttributeError: 'str' object has no attribute 'iter_parents'
art049 commented 7 months ago

I think this issue is related to https://github.com/pytest-dev/pytest/pull/11785 Having a look rn

davidhewitt commented 7 months ago

Thanks! 🎉

art049 commented 7 months ago

No problem :)