Eliminate double book keeping in `extern`, just using `_vendor` directly

[abravalheri]

Main idea:

• Use _vendor directly instead of extern

This is an investigation on https://github.com/pypa/setuptools/pull/4320#pullrequestreview-2014802225

Summary of changes

Intentional changes:

• 836168ba18a743697cb9b9490afb3e87facde438: Modify tool/vendored.py so that: <------------------- this is possibly the most relevant commit for review

  1. Transitive dependencies are automatically documented (so we don't loose track of what needs what).
     • This is done via pip-compile and vendored.in (direct dependencies) ==> vendored.txt. See e09f4ae0188c0a7d44cd3c0b1478a867a540c37b.
  2. Instead of a case-by-case import rewrite, all vendored imports are rewritten at once
     • Case-by-case workarounds were kept (e.g. intentionally delayed imports)
  3. Sync versions of vendored packages in pkg_resources and setuptools so that it is easier for "de-vendoring" (this is done via --constraint in pip).

• 2c4590655537439a975d41228766a6c6e8906833: Remove setuptools.extern and pkg_resouces.extern.

Collateral changes:

• b0dcf0d1fea38e5fff5280a5e40fdb9e711f067b: Update on vendored packages
• 9e7decef9707336b169106cb5eabf4cf1e9a557f: Replace extern imports with _vendor

Changes that were needed just to make the tests pass:

• 96e1d9f00689a6a2f216be278a2f9afb071aa3e9: Changes to make mypy happy (there is one actual fix).
• 9abbdb928182612fbf37ae024dece543f1f3b02b: Temporary workaround for https://github.com/pytest-dev/pytest/issues/12254.

Closes

Pull Request Checklist

• [ ] Changes have tests
• [ ] News fragment added in newsfragments/. (See [documentation][PR docs] for details)

[PR docs]: https://setuptools.pypa.io/en/latest/development/developer-guide.html#making-a-pull-request

---

Drawbacks

The separation between the direct dependencies (vendored.in) and the automatic logging and documentation of the transitive dependencies (vendored.txt) is nice from some point-of-view.

However, it also requires that we run tox -e vendor using the lowest version of Python supported by setuptools (otherwise conditional environment markers may be ignored). This may be easy to forget.

[abravalheri]

NOTE:

There are some people out there importing extern: https://github.com/search?q=%2Ffrom+setuptools.extern%2F+language%3APython&type=code&l=Python

Maybe we should have a deprecation here? Although, I find it weird that a 3rd-party is to importing extern without at least a try-except.

[abravalheri]

@jaraco is this an approach you would be interested for us to follow in setuptools?

[jaraco]

Although, I find it weird that a 3rd-party is to importing extern without at least a try-except.

IIRC, my intention for extern was that it was providing the compatibility shim - if the vendored packages were present, they'd be imported, but if they were missing, the dependencies could be loaded from sys.path. That's what VendorImporter.search_path did. I wanted to make it trivially easy for a downstream distro to simply remove the _vendor directory and supply the dependencies and everything would just work. Hard-coding _vendor in the imports will break this assumption (and require distro packagers to re-write all of these imports).

My hope/expectation was that at some point the vendored packages would be removed in all but a few isolated scenarios and real dependencies would be used. That vision never panned out.

Maybe we should have a deprecation here?

Although I made extern a public-facing name, I never intended it to be used for anything except project specifically dependent on setuptools' dependencies (e.g. they need to catch the same exception from packaging). I never considered the backward compatibility implications here.

@jaraco is this an approach you would be interested for us to follow in setuptools?

My hope was that setuptools (and maybe pkg_resources) could simply import these dependencies instead of hard-coding the imports from their vendored location. I was thinking something like:

- from setuptools.extern import platformdirs
+ import platformdirs

And then early in the import of setuptools ensure that everything in setuptools._vendor appears in sys.path (such that the packages always prefer installed packages and only fall back to _vendor in bootstrap scenarios). I even have hopes to make the vendoring an exceptional case, where they're only installed (or added to sys.path) under special conditions that are known to need special bootstraping support, but otherwise rely entirely on standard packaging resolution. That is, under most circumstances, a PEP 517/518 installer will ensure dependencies are met before doing anything with setuptools... and only in source-only builds would there need to be a bootstrap step to break the cyclic dependency (e.g. setuptools --> packaging (from source dist) --> setuptools).

I'm not confident that transitioning to a hard-coded _vendor import will get us there.