Bandit broken via `stevedore` dependency with `importlib-metadata>=5`.

[emcd]

Describe the bug

Bandit depends on the stevedore package for extension management. That package, in turn, depends on importlib-metadata for the entrypoints mechanism that it uses. importlib-metadata 5.0.0 was recently released and apparently removed compatibility with the mechanism used by stevedore.

The following crash now happens, if importlib-metadata 5.0.0 is installed:

$ bandit --version
Traceback (most recent call last):
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/stevedore/_cache.py", line 159, in _get_data_for_path
    with open(filename, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/me/.cache/python-entrypoints/19c8187c80a898f0d56eed9b81a44d91389b2a2953186081f4546ac88dcb6e00'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/me/src/python-lockup/stevedore-bug/bin/bandit", line 5, in <module>
    from bandit.cli.main import main
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/bandit/__init__.py", line 7, in <module>
    from bandit.core import config  # noqa
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/bandit/core/__init__.py", line 5, in <module>
    from bandit.core import config  # noqa
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/bandit/core/config.py", line 15, in <module>
    from bandit.core import extension_loader
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/bandit/core/extension_loader.py", line 109, in <module>
    MANAGER = Manager()
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/bandit/core/extension_loader.py", line 21, in __init__
    self.load_formatters(formatters_namespace)
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/bandit/core/extension_loader.py", line 29, in load_formatters
    verify_requirements=False,
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/stevedore/extension.py", line 136, in __init__
    verify_requirements)
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/stevedore/extension.py", line 218, in _load_plugins
    for ep in self.list_entry_points():
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/stevedore/extension.py", line 207, in list_entry_points
    eps = list(_cache.get_group_all(self.namespace))
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/stevedore/_cache.py", line 179, in get_group_all
    data = self._get_data_for_path(path)
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/stevedore/_cache.py", line 162, in _get_data_for_path
    data = _build_cacheable_data(path)
  File "/home/me/src/python-lockup/stevedore-bug/lib/python3.7/site-packages/stevedore/_cache.py", line 110, in _build_cacheable_data
    for name, group_data in real_groups.items():
AttributeError: 'EntryPoints' object has no attribute 'items'

Reproduction steps

For crash:

1. ASDF_PYTHON_VERSION=3.7.14 python3 -m venv stevedore-bug # or your favorite way of creating a Python virtual environment
2. . stevedore-bug/bin/activate # Assuming a Bourne shell-like sourcing mechanism for your shell.
3. pip install bandit importlib-metadata
4. bandit --version # crashes

To prove that newest importlib-metadata is the culprit by constraining it:

1. ASDF_PYTHON_VERSION=3.7.14 python3 -m venv stevedore-workaround
2. . stevedore-workaround/bin/activate
3. pip install bandit 'importlib-metadata<5'
4. bandit --version # works

Expected behavior

Display of Bandit version:

bandit 1.7.4
  python version = 3.7.14 (default, Sep 12 2022, 16:42:27) [GCC 9.4.0]

Bandit version

1.7.4 (Default)

Python version

3.7

Additional context

Have not reported upstream to stevedore project yet since their issue tracker is not on Github. May do so later, depending on level of effort/hassle. Probably best workaround for now is to explicitly clamp the importlib-metadata version.

[emcd]

pip freeze shows that stevedore==3.5.0 is installed on Python 3.7. stevedore>=4 has a fix in it, but it is only supported on Python 3.8 and higher. Bandit says that it still supports Python 3.7, so I guess the onus is on Bandit to decide how to handle the Python 3.7 case with stevedore.

[mportesdev]

Hi @emcd thanks for pointing this out. The problem has already been reported at https://bugs.launchpad.net/python-stevedore/+bug/1991559

I wonder if a fix will be incorporated in stevedore 3.x as well as 4.x

[emcd]

Thanks for the update @mportesdev. I wasn't sure if it was worth reporting upstream, since it looked like they didn't care about Python 3.7 anymore. But, I'm glad someone else did report it. I see that they have ported the fix to various code-named OpenStack releases (wallaby, xena, yoga, zed); hopefully that will include a backport for stevedore 3.x (and hence Python 3.7).

[kszmigiel]

@emcd thank you for creating this issue, I was tearing my hair out since yesterday as I couldn't find the reason behind this. Do you think updating the whole app to python 3.8 will help? Unfortunately I can't wait for official fix, and can't skip bandit check either.

[mportesdev]

@kszmigiel Until a fix is released, you can pip install 'importlib-metadata<5;python_version<"3.8"' manually in the environment where bandit is installed.

[emcd]

@kszmigiel : I agree with @mportesdev. No need to upgrade to Python 3.8 for this. You could just put importlib-metadata<5 into your requirements.txt or equivalent until upstream fixes or Michal's patch is applied to Bandit. This will give you consistency across all your versions of Python, since it is unlikely that anything actually requires importlib-metadata >= 5 yet.

[emcd]

stevedore==3.5.1 was released upstream yesterday and contains a fix for this bug on Python 3.7. Closing the issue.

[mcdonnnj]

stevedore==3.5.1 was released upstream yesterday and contains a fix for this bug on Python 3.7. Closing the issue.

@emcd I'm seeing a failure to load tests when running on Python 3.7 on stevedore 3.5.1 (new release). Are you seeing expected functionality with that release on importlib-metadata 5.0.0?

$ python3 --version
Python 3.7.15
$ pip list | grep "bandit\|importlib-metadata\|stevedore"
bandit               1.7.4
importlib-metadata   5.0.0
stevedore            3.5.1
$ bandit -r .
[main]  INFO    profile include tests: None
[main]  INFO    profile exclude tests: None
[main]  INFO    cli include tests: None
[main]  INFO    cli exclude tests: None
[main]  INFO    running on Python 3.7.15
[main]  ERROR   No tests would be run, please check the profile.

[emcd]

stevedore==3.5.1 was released upstream yesterday and contains a fix for this bug on Python 3.7. Closing the issue.

@emcd I'm seeing a failure to load tests when running on Python 3.7 on stevedore 3.5.1 (new release). Are you seeing expected functionality with that release on importlib-metadata 5.0.0?

Good catch, @mcdonnnj . When I tested earlier, I was in a hurry and saw that bandit --version no longer crashed and thought we were good. But, I can reproduce the same problem as you. Definitely looks like a bug. Thanks for opening a new issue. I will upvote.