[FR] Allow ignoring ImportError (or making it non-fatal) for setuptools plugins

What's the problem this feature will solve?

The current setup for setuptools plugins can be challenging for us in Gentoo because setuptools will try to load all plugins even if no package is known to need it yet.

Modelling the relationship between setuptools, its plugins, and plugin dependencies is non-trivial. setuptools doesn't depend on the plugins, but if a plugin is installed, it has to always be in a usable state and setuptools effectively starts to depend on it.

In Gentoo, Python is "slotted" so it can be installed in parallel with other versions of Python. This is applied to Python libraries too. When migrating between Python versions, we sometimes have a problem.

For example, setuptools, setuptools-rust, and semantic_version might be installed for Python 3.11. When moving to Python 3.12, setuptools must be installed first. If semantic_version is rebuilt for Python 3.12, temporarily breaking setuptools-rust, any package which tries to use setuptools will be broken because of the temporarily-broken plugin, even if that package doesn't use setuptools-rust.

We suspect it could manifest with pip too given pip either didn't in the past or doesn't enforce dependency constraints, possibly by upgrading a dependency of a plugin to an incompatible version.

Describe the solution you'd like

It would help us enormously if setuptools either default-ignored ImportError on plugins and warned on it instead, or allowed controlling this via e.g. an environment variable.

Alternative Solutions

We have considered making setuptools depend on all plugin providers in Gentoo but this would pull in many dependencies and be rather unwieldy. It also wouldn't work if users package their own plugins.

Additional context

This came up downstream in Gentoo at https://bugs.gentoo.org/933553 (and in the past at https://bugs.gentoo.org/663324).

Code of Conduct

[X] I agree to follow the PSF Code of Conduct

cc @mgorny @zmedico @eli-schwartz

This is hard to explain, so let me try adding some context.

A single PyPI package is mapped into a single Gentoo package. This package installs the relevant Python package for one or more Python versions, with users being able to select versions they need. Our package manager normally enforces dependencies in such a way that e.g. if A depends on B, then B needs to built with the same Python version as A first. However, this can't always work fine.

For example, consider the following dependency graph:

setuptools-rust depends on setuptools and semantic-version
foo depends on setuptools

Note that there is no dependency between foo and setuptools-rust.

Initial state A is that all four packages were built for Python 3.11.

Desired state B is that:

setuptools-rust and semantic-version are built for 3.12
foo is built for 3.11
setuptools is built for 3.11+3.12

Now, the problem is that the package may order the builds as follows:

Build setuptools for 3.11+3.12
Build semantic-version for 3.12
Build foo for 3.11
Build setuptools-rust for 3.12

(the ordering is weird but apparently it's caused by some more dependencies between otehr packages but I don't want to add too much complexity here)

Now, the problem is that at 3. setuptools-rust is built for 3.11 but semantic-version not anymore, so setuptools-rust is effectively broken. Technically, this should be fine since neither foo nor setuptools depend on setuptools-rust. However, because of the implicit plugin loading, setuptools nevertheless tries to load it and fails.

Slightly more interesting, my original reproducer actually had this:

Build semantic-version for 3.12

Build setuptools for 3.11+3.12

Build foo for 3.11

Build setuptools-rust for 3.12

As a result it was setuptools that failed to build:

  File "/var/tmp/portage/dev-python/setuptools-70.0.0/work/setuptools-70.0.0/setuptools/build_meta.py", line 410, in build_wheel
    return self._build_with_temp_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/setuptools-70.0.0/work/setuptools-70.0.0/setuptools/build_meta.py", line 395, in _build_with_temp_dir
    self.run_setup()
  File "/var/tmp/portage/dev-python/setuptools-70.0.0/work/setuptools-70.0.0/setuptools/build_meta.py", line 311, in run_setup
    exec(code, locals())
  File "<string>", line 93, in <module>
  File "/var/tmp/portage/dev-python/setuptools-70.0.0/work/setuptools-70.0.0/setuptools/__init__.py", line 103, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/setuptools-70.0.0/work/setuptools-70.0.0/setuptools/_distutils/core.py", line 146, in setup
    _setup_distribution = dist = klass(attrs)
                                 ^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/setuptools-70.0.0/work/setuptools-70.0.0/setuptools/dist.py", line 307, in __init__
    _Distribution.__init__(self, dist_attrs)
  File "/var/tmp/portage/dev-python/setuptools-70.0.0/work/setuptools-70.0.0/setuptools/_distutils/dist.py", line 284, in __init__
    self.finalize_options()
  File "/var/tmp/portage/dev-python/setuptools-70.0.0/work/setuptools-70.0.0/setuptools/dist.py", line 658, in finalize_options
    for ep in sorted(loaded, key=by_order):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/portage/dev-python/setuptools-70.0.0/work/setuptools-70.0.0/setuptools/dist.py", line 657, in <lambda>
    loaded = map(lambda e: e.load(), filtered)
                           ^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.11/site-packages/setuptools_rust/__init__.py", line 1, in <module>
    from .build import build_rust
  File "/usr/lib/python3.11/site-packages/setuptools_rust/build.py", line 28, in <module>
    from .command import RustCommand
  File "/usr/lib/python3.11/site-packages/setuptools_rust/command.py", line 7, in <module>
    from .extension import RustExtension
  File "/usr/lib/python3.11/site-packages/setuptools_rust/extension.py", line 21, in <module>
    from semantic_version import SimpleSpec
ModuleNotFoundError: No module named 'semantic_version'

pypa / setuptools