pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.97k stars 2.66k forks source link

v8.1.0 changes plugin registration #12061

Closed glatterf42 closed 7 months ago

glatterf42 commented 7 months ago

Thanks for your work and effort in maintaining pytest! Unfortunately, the release of v8.1.0 seems to break all our test suites (e.g. at ixmp and message_ix. I've tried to confirm this by updating just pytest locally in my venv and this did indeed break the tests locally as well. So I had:

fridolin@fridolin-Latitude-5520 ~/ixmp (main)> (mix312) pip list
Package                       Version                                Editable project location
----------------------------- -------------------------------------- --------------------------------
alabaster                     0.7.16
anyio                         4.2.0
argon2-cffi                   23.1.0
argon2-cffi-bindings          21.2.0
arrow                         1.3.0
asttokens                     2.4.1
async-lru                     2.0.4
asyncssh                      2.14.2
attrs                         23.2.0
Babel                         2.14.0
beautifulsoup4                4.12.3
bleach                        6.1.0
bottle                        0.12.25
cattrs                        23.2.3
certifi                       2023.11.17
cffi                          1.16.0
charset-normalizer            3.3.2
click                         8.1.7
cloudpickle                   3.0.0
colorama                      0.4.6
comm                          0.2.1
contourpy                     1.2.0
coverage                      7.4.1
cryptography                  42.0.2
cycler                        0.12.1
dask                          2024.1.1
debugpy                       1.8.0
decorator                     5.1.1
defusedxml                    0.7.1
docutils                      0.20.1
et-xmlfile                    1.1.0
execnet                       2.0.2
executing                     2.0.1
fastjsonschema                2.19.1
fonttools                     4.47.2
fqdn                          1.5.1
fsspec                        2023.12.2
genno                         1.23.0
gitdb                         4.0.11
GitPython                     3.1.41
graphviz                      0.20.1
iam-units                     2023.9.12
idna                          3.6
imagesize                     1.4.1
importlib-metadata            7.0.1
iniconfig                     2.0.0
ipykernel                     6.29.0
ipython                       8.21.0
ipywidgets                    8.1.1
isoduration                   20.11.0
ixmp                          3.8.1.dev9+g9b93531                    /home/fridolin/ixmp
JayDeBeApi                    1.2.3
jedi                          0.19.1
Jinja2                        3.1.3
joblib                        1.3.2
JPype1                        1.5.0
json5                         0.9.14
jsonpointer                   2.4
jsonschema                    4.21.1
jsonschema-specifications     2023.12.1
jupyter                       1.0.0
jupyter_client                8.6.0
jupyter-console               6.6.3
jupyter_core                  5.7.1
jupyter-events                0.9.0
jupyter-lsp                   2.2.2
jupyter_server                2.12.5
jupyter_server_terminals      0.5.2
jupyterlab                    4.0.12
jupyterlab_pygments           0.3.0
jupyterlab_server             2.25.2
jupyterlab-widgets            3.0.9
kiwisolver                    1.4.5
latexcodec                    2.0.1
locket                        1.0.0
lxml                          5.1.0
MarkupSafe                    2.1.4
matplotlib                    3.8.2
matplotlib-inline             0.1.6
memory-profiler               0.61.0
message_data                  2020.6.21.dev2803+g4b651605c.d20240201 /home/fridolin/message_data
message_ix                    3.8.1.dev4+g4dae41e                    /home/fridolin/message_ix
message-ix-models             2023.11.25.dev61+g99244af2             /home/fridolin/message-ix-models
mistune                       3.0.2
mizani                        0.9.3
mypy                          1.8.0
mypy-extensions               1.0.0
nbclient                      0.9.0
nbconvert                     7.14.2
nbformat                      5.9.2
nest-asyncio                  1.6.0
notebook                      7.0.7
notebook_shim                 0.2.3
numpy                         1.26.3
numpydoc                      1.6.0
openpyxl                      3.1.0
oracledb                      2.0.1
overrides                     7.7.0
packaging                     23.2
pandas                        2.2.0
pandocfilters                 1.5.1
parso                         0.8.3
partd                         1.4.1
patsy                         0.5.6
pexpect                       4.9.0
pillow                        10.2.0
Pint                          0.17
pip                           23.3.2
platformdirs                  4.2.0
plotnine                      0.12.4
pluggy                        1.4.0
pooch                         1.8.0
pprint36                      3.9.16.0
pretenders                    1.4.5
prometheus-client             0.19.0
prompt-toolkit                3.0.43
psutil                        5.9.8
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    9.0.0
pyam-iamc                     1.3.1
pyarrow                       15.0.0
pybtex                        0.24.0
pybtex-docutils               1.0.3
pycountry                     23.12.11
pycparser                     2.21
Pygments                      2.17.2
pyparsing                     3.1.1
pytest                        8.0.0
pytest-benchmark              4.0.0
pytest-cov                    4.1.0
pytest-rerunfailures          13.0
pytest-xdist                  3.5.0
python-dateutil               2.8.2
python-json-logger            2.0.7
pytz                          2023.4
PyYAML                        6.0.1
pyzmq                         25.1.2
qtconsole                     5.5.1
QtPy                          2.4.1
referencing                   0.33.0
requests                      2.31.0
requests-cache                1.1.1
requests-ftp                  0.3.1
rfc3339-validator             0.1.4
rfc3986-validator             0.1.1
rpds-py                       0.17.1
ruff                          0.1.15
scipy                         1.12.0
sdmx1                         2.13.1
seaborn                       0.13.2
Send2Trash                    1.8.2
setuptools                    69.0.3
setuptools-scm                8.0.4
six                           1.16.0
smmap                         5.0.1
sniffio                       1.3.0
snowballstemmer               2.2.0
soupsieve                     2.5
Sphinx                        7.2.6
sphinx-rtd-theme              2.0.0
sphinxcontrib-applehelp       1.0.8
sphinxcontrib-bibtex          2.6.2
sphinxcontrib-devhelp         1.0.6
sphinxcontrib-htmlhelp        2.0.5
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.7
sphinxcontrib-serializinghtml 1.1.10
stack-data                    0.6.3
statsmodels                   0.14.1
tabulate                      0.9.0
teamcity-messages             1.32
terminado                     0.18.0
tinycss2                      1.2.1
toolz                         0.12.1
tornado                       6.4
tqdm                          4.66.1
traitlets                     5.14.1
transport-energy              2021.5.4
types-python-dateutil         2.8.19.20240106
types-tqdm                    4.66.0.20240106
typing_extensions             4.9.0
tzdata                        2023.4
uri-template                  1.3.0
url-normalize                 1.4.3
urllib3                       2.2.0
wcwidth                       0.2.13
webcolors                     1.13
webencodings                  0.5.1
websocket-client              1.7.0
wheel                         0.42.0
widgetsnbextension            4.0.9
xarray                        2024.1.1
xlrd                          2.0.1
XlsxWriter                    3.1.9
zipp                          3.17.0

[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: pip install --upgrade pip
fridolin@fridolin-Latitude-5520 ~/ixmp (main)> (mix312) pytest ixmp/tests/test_config.py 
=========================================================== test session starts ============================================================
platform linux -- Python 3.12.2, pytest-8.0.0, pluggy-1.4.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
ixmp config: Values(platform={'default': 'local', 'local': {'class': 'jdbc', 'driver': 'hsqldb'}}, message_buildings_dir=None)
rootdir: /home/fridolin/ixmp
configfile: pyproject.toml
plugins: xdist-3.5.0, cov-4.1.0, benchmark-4.0.0, teamcity-messages-1.32, anyio-4.2.0, rerunfailures-13.0
collected 8 items                                                                                                                          

ixmp/tests/test_config.py ........                                                                                                   [100%]

============================================================= warnings summary =============================================================
../.virtualenvs/mix312/lib/python3.12/site-packages/dateutil/tz/tz.py:37
  /home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/dateutil/tz/tz.py:37: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
    EPOCH = datetime.datetime.utcfromtimestamp(0)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================= 8 passed, 1 warning in 0.21s =======================================================
fridolin@fridolin-Latitude-5520 ~/ixmp (main)> (mix312) pip install --upgrade pytest
Requirement already satisfied: pytest in /home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages (8.0.0)
Collecting pytest
  Downloading pytest-8.1.0-py3-none-any.whl.metadata (7.6 kB)
Requirement already satisfied: iniconfig in /home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages (from pytest) (2.0.0)
Requirement already satisfied: packaging in /home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages (from pytest) (23.2)
Requirement already satisfied: pluggy<2.0,>=1.4 in /home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages (from pytest) (1.4.0)
Downloading pytest-8.1.0-py3-none-any.whl (335 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 335.5/335.5 kB 4.2 MB/s eta 0:00:00
Installing collected packages: pytest
  Attempting uninstall: pytest
    Found existing installation: pytest 8.0.0
    Uninstalling pytest-8.0.0:
      Successfully uninstalled pytest-8.0.0
Successfully installed pytest-8.1.0

And this lead to the following error and traceback:

fridolin@fridolin-Latitude-5520 ~/ixmp (main)> (mix312) pytest ixmp/tests/test_config.py
Traceback (most recent call last):
  File "/home/fridolin/.virtualenvs/mix312/bin/pytest", line 8, in <module>
    sys.exit(console_main())
             ^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 195, in console_main
    code = main()
           ^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 153, in main
    config = _prepareconfig(args, plugins)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 335, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_callers.py", line 138, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/helpconfig.py", line 105, in pytest_cmdline_parse
    config = yield
             ^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1141, in pytest_cmdline_parse
    self.parse(args)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1490, in parse
    self._preparse(args, addopts=addopts)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1394, in _preparse
    self.hook.pytest_load_initial_conftests(
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__
    return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_callers.py", line 138, in _multicall
    raise exception.with_traceback(exception.__traceback__)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/warnings.py", line 150, in pytest_load_initial_conftests
    return (yield)
            ^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_callers.py", line 121, in _multicall
    teardown.throw(exception)  # type: ignore[union-attr]
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/capture.py", line 153, in pytest_load_initial_conftests
    yield
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
    res = hook_impl.function(*args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 1219, in pytest_load_initial_conftests
    self.pluginmanager._set_initial_conftests(
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 577, in _set_initial_conftests
    self._try_load_conftest(
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 615, in _try_load_conftest
    self._loadconftestmodules(
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 655, in _loadconftestmodules
    mod = self._importconftest(
          ^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 731, in _importconftest
    self.consider_conftest(mod, registration_name=conftestpath_plugin_name)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 812, in consider_conftest
    self.register(conftestmodule, name=registration_name)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 508, in register
    self.consider_module(plugin)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 820, in consider_module
    self._import_plugin_specs(getattr(mod, "pytest_plugins", []))
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 827, in _import_plugin_specs
    self.import_plugin(import_spec)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 864, in import_plugin
    self.register(mod, modname)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/_pytest/config/__init__.py", line 497, in register
    plugin_name = super().register(plugin, name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_manager.py", line 167, in register
    self._verify_hook(hook, hookimpl)
  File "/home/fridolin/.virtualenvs/mix312/lib/python3.12/site-packages/pluggy/_manager.py", line 342, in _verify_hook
    raise PluginValidationError(
pluggy._manager.PluginValidationError: Plugin 'ixmp.testing' for hook 'pytest_report_header'
hookimpl definition: pytest_report_header(config, startdir)
Argument(s) {'startdir'} are declared in the hookimpl but can not be found in the hookspec

And to confirm

fridolin@fridolin-Latitude-5520 ~/ixmp (main) [1]> (mix312) pip list | grep pytest
pytest                        8.1.0
pytest-benchmark              4.0.0
pytest-cov                    4.1.0
pytest-rerunfailures          13.0
pytest-xdist                  3.5.0

Locally, I'm running this on Ubuntu 22.04 with Python 3.12, but as I said above, the exact same error occurs on all OS and from Python 3.8 through 3.12 in our CI.

The version of pluggy hasn't changed (neither on CI nor locally), same is true for ixmp, though that might be ill-configured so that the plugin is not properly registered. If that is the case, please let me know what we need to change on our end to accommodate v8.1.0 :)

The-Compiler commented 7 months ago

See https://docs.pytest.org/en/8.0.x/deprecations.html#py-path-local-arguments-for-hooks-replaced-with-pathlib-path - i.e. ixmp needs to use start_path and not startdir.

robsan00 commented 7 months ago

I just ran into this as well. Apparently, with version 7.0.0, startdir got an alternative name start_path, the idea being that startdir is getting deprecated. If you look deeper into the 7.0.0 changelog, you can find an indication about the deprecation (https://docs.pytest.org/en/stable/deprecations.html#py-path-local-arguments-for-hooks-replaced-with-pathlib-path). Not sure why this was thrown out for 8.1.0 though, it's not listed in the changelog and isn't the idea to do backwards incompatible changes only in major version changes? The docu marks this as having been removed in 8.0, which is not the case.

glatterf42 commented 7 months ago

Thanks, both :)

bluetech commented 7 months ago

@robsan00 pytest's policy is add deprecation warning -> change deprecation warning to error by default on MAJOR.0 release -> completely remove on MAJOR.1 release. This is to give people one last chance to handle the deprecations. Probably your project suppresses warnings?

khaeru commented 7 months ago

pytest's policy is add deprecation warning -> change deprecation warning to error by default on MAJOR.0 release -> completely remove on MAJOR.1 release. This is to give people one last chance to handle the deprecations. Probably your project suppresses warnings?

Policy notwithstanding, this issue arose because no such warning was ever added; only the mention in the change log. You can confirm this with the following in an empty directory:

conftest.py:

def pytest_report_header(config, startdir):  # Old-style signature
    print("Hello, world!")

test_all.py:

def test_nothing():
   pass

With 7.4.4 <= pytest < 8.1, this runs and no warning is emitted. With 8.1.0, it errors.

Maybe it is technically infeasible or prohibitively complex for pytest to introspect every hookimpl and raise warnings; that's fair. Also fair to expect users to read the change log carefully once in a while. But one can't fault users for disregarding a non-existent warning.

RonnyPfannschmidt commented 7 months ago

i suspect we have a issue with pytest hooking into warnings, its possible the setup for warning capture is not completed at that point in time

bluetech commented 7 months ago

@khaeru I think you are right, it seems that the warnings were issued when the hook is called with the deprecated arguments, but not when a hookimpl is defined accepting deprecated arguments. This is an omission that I didn't realize myself.