search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.97k stars 2.66k forks source link

8.1.1 raises exception if `boto` is imported in environment #12179

Closed effigies closed 6 months ago

effigies commented 6 months ago

Pytest 8.1.1 on Ubuntu 22.04.

❯ pipenv run pytest ./tests --pdb
============================================= test session starts =============================================
platform linux -- Python 3.10.10, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/chris/Projects/openneuro/openneuro/services/datalad
configfile: pytest.ini
plugins: xdist-3.5.0, cov-4.1.0
collecting ... 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stderr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[WARNING] Requested extension 'next' is not available 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/runner.py:340: in from_call
    result: Optional[TResult] = func()
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/runner.py:388: in collect
    return list(collector.collect())
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:576: in collect
    self._register_setup_module_fixture()
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:589: in _register_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:315: in obj
    self._obj = obj = self._getobj()
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:573: in _getobj
    return importtestmodule(self.path, self.config)
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:520: in importtestmodule
    mod = import_path(
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/pathlib.py:546: in import_path
    mod = _import_module_using_spec(
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/pathlib.py:630: in _import_module_using_spec
    spec = meta_importer.find_spec(module_name, [str(module_location)])
E   AttributeError: '_SixMetaPathImporter' object has no attribute 'find_spec'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/pathlib.py(630)_import_module_using_spec()
-> spec = meta_importer.find_spec(module_name, [str(module_location)])
(Pdb) meta_importer
<boto.vendored.six._SixMetaPathImporter object at 0x76dad9949d20>
(Pdb) l
625             "src_tests_test_foo".
626         """
627         # Checking with sys.meta_path first in case one of its hooks can import this module,
628         # such as our own assertion-rewrite hook.
629         for meta_importer in sys.meta_path:
630  ->         spec = meta_importer.find_spec(module_name, [str(module_location)])
631             if spec is not None:
632                 break
633         else:
634             spec = importlib.util.spec_from_file_location(module_name, str(module_path))
635         if spec is not None:

Observed in https://github.com/OpenNeuroOrg/openneuro/pull/3020, which attempted to auto-update dependencies.

$ pip list
Package            Version
------------------ ---------------------
annexremote        1.6.4
async-timeout      4.0.3
boto               2.49.0
boto3              1.34.75
botocore           1.34.75
certifi            2024.2.2
cffi               1.16.0
chardet            5.2.0
charset-normalizer 3.3.2
coverage           7.4.4
cryptography       42.0.5
datalad            0.19.6
Deprecated         1.2.14
distro             1.9.0
dnspython          2.6.1
ecs-logging        2.1.0
elastic-apm        6.21.4.post8347027212
elastic-transport  8.13.0
elasticsearch      8.13.0
exceptiongroup     1.2.0
execnet            2.0.2
falcon             3.1.3
falcon-elastic-apm 0.1.1
fasteners          0.19
gevent             24.2.1
gitdb              4.0.11
GitPython          3.1.43
greenlet           3.0.3
gunicorn           21.2.0
humanize           4.9.0
idna               3.6
importlib_metadata 7.1.0
iniconfig          2.0.0
iso8601            2.1.0
jaraco.classes     3.4.0
jaraco.context     4.3.0
jaraco.functools   4.0.0
jeepney            0.8.0
jmespath           1.0.1
keyring            25.0.0
keyrings.alt       5.0.1
looseversion       1.3.0
mock               5.1.0
more-itertools     10.2.0
msgpack            1.0.8
packaging          24.0
patool             2.2.0
pip                24.0
platformdirs       4.2.0
pluggy             1.4.0
pycparser          2.22
pygit2             1.14.1
PyGithub           2.2.0
PyJWT              2.8.0
PyNaCl             1.5.0
pytest             8.1.1
pytest-cov         4.1.0
pytest-xdist       3.5.0
python-dateutil    2.9.0.post0
python-gitlab      4.4.0
redis              5.0.3
requests           2.31.0
requests-toolbelt  1.0.0
s3transfer         0.10.1
SecretStorage      3.3.3
setuptools         69.2.0
six                1.16.0
smmap              5.0.1
tomli              2.0.1
tqdm               4.66.2
typing_extensions  4.10.0
urllib3            2.2.1
wheel              0.42.0
wrapt              1.14.1
zipp               3.18.1
zope.event         5.0
zope.interface     6.2
The-Compiler commented 6 months ago

Anything that's on the sys.meta_path can/should have a find_spec ever since Python 3.4 (2013). It not being there is deprecated since Python 3.10 (2021), and indeed six added this back in 2021.

boto is using a vendored six version from 2018, so you most likely want to switch to boto3 instead which is actively maintained (as also recommended by boto, which hasn't been touched in the past 2 years).

Not sure if pytest should try to paint over this. If I'm not mistaken, this code was around since pytest 6 in 2020 (ab6dacf1d1e1ff0c5be70a3c5f48e63168168721), and I don't think the recent move (4dea18308bafafb0be0b2d07e02328051c3095d5) changed anything?

effigies commented 6 months ago

I don't disagree, but I have very limited control over when a dependency upgrades from boto to boto3. I am using this to encourage them to prioritize it. I haven't looked into what change in 8.1 caused this; possibly you removed a suppression of this error, but it does seem like a good idea to guard against a bad importer that some other library stuck into sys.meta_path.

Zac-HD commented 6 months ago

Happy to say that we're not going to "fix" this, and your other dependency should.