search

pypa / setuptools

Official project repository for the Setuptools build system
https://pypi.org/project/setuptools/
MIT License
2.36k stars 1.15k forks source link

Adopt distutils #417

Closed ghost closed 3 years ago

ghost commented 8 years ago

Originally reported by: jaraco (Bitbucket: jaraco, GitHub: jaraco)

Setuptools builds on the distutils framework of the Python Stdlib. Doing so has benefits of depending on a standard, extensible framework, but also has downsides, such as limiting the advancement of distutils (across many Python versions) but simultaneously adding monkey patches to patch bugs in distutils.

I'd like to consider if Setuptools should adopt distutils (or a version of it). Doing so would decouple Setuptools from distutils, freeing Setuptools from the constraints of distutils and freeing distutils from the expectations of Setuptools.

Doing so would allow distutils to adapt at the same pace as Setuptools, addressing issues and vetting patches that could be ported back to distutils in CPython.

I've once proposed this idea and it was promptly dismissed by @RonnyPfannschmidt. Why is vendoring distutils in Setuptools not an option?

ghost commented 8 years ago

Original comment by andi0661 (Bitbucket: andi0661, GitHub: Unknown):

I like the idea of separating distutils and setuptools, but given @RonnyPfannschmidt 's and @ralfgommers 's comments, it seems to me that there is so much new legacy on top of setuptools that it would require a mechanism that allows for incompatible changes. Maybe a new major version is sufficient, or a different package name. That would allow packages to transition to it as they have a need for the better functions, less bugs, etc.

ghost commented 8 years ago

Original comment by ralfgommers (Bitbucket: ralfgommers, GitHub: Unknown):

I don't quite understand how the "vendoring and put in a new namespace" idea would help. There are other packages that extend (and monkeypatch) distutils. So they'd then all have to change to instead extend/monkeypatch setuptools.distutils?

ghost commented 8 years ago

Original comment by RonnyPfannschmidt (Bitbucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

i fear this will result in a certain hell, there is no reasonable way to get rid of the current legacy within setuptools without creating many lines of compatibility breaks

ghost commented 8 years ago

Original comment by ncoghlan (Bitbucket: ncoghlan, GitHub: ncoghlan):

Python does depend on distutils to build some extension modules, but not the core interpreter itself.

However, unbundling per se probably isn't worth the hassle, as implementing the 1/2/3 approach I suggested above based on a "setuptool._distutils" vendored copy should provide most of the benefits with a fraction of the pain. The key will be ensuring the stdlib distutils is kept in sync with the setuptools vendored copy for bug fixes (all versions) and new features (future versions).

ghost commented 8 years ago

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

My thinking was more along the lines with what Marcus is suggesting.

Thanks to Nick's comments, I'm reminded that pip install (of non-binary distributions) and also setup.py install (where setuptools is somehow imported) are expected to run with setuptools' overrides. Maintaining that expectation could be tricky, but is probably achievable.

ghost commented 8 years ago

Original comment by qwcode (Bitbucket: qwcode, GitHub: qwcode):

I love the possibility. btw, I recall this being discussed a couple years back. At the time, I guess it was too radical to gain any traction.

as for Nick's 1/2/3 points, wouldn't vendoring it, effectively give it a new namespace, and avoid any concerns with conflicts?

ghost commented 8 years ago

Original comment by dstufft (Bitbucket: dstufft, GitHub: dstufft):

@ncoghlan Would that work? Doesn't Python depend on distutils to build itself?

ghost commented 8 years ago

Original comment by ncoghlan (Bitbucket: ncoghlan, GitHub: ncoghlan):

I think this could work well with an eventual approach like the following:

  1. By default, importing setuptools would leave the standard distutils namespace alone
  2. setuptools would gain a "setuptools.override_distutils()" command that injected the vendored distutils into sys.modules (and complained if the standard library one had already been imported)
  3. pip was updated to both import setuptools and tell it to explicitly override distutils

There'd need to be a transition period where setuptools did the injection by default, as the monkeypatching of distutils is currently implicit in importing setuptools

Longer term, we could eventually aim to move distutils out of the normal standard library update cycle, and instead install it via ensurepip (such that it evolves at the pace of the Python packaging ecosystem, rather than at the pace of the underlying language runtime)

jaraco commented 6 years ago

@xoviat : To answer your question from the other ticket, I'd very much appreciate a PR.

ghost commented 6 years ago

Okay, I'll send one your way sometime soon.

ghost commented 6 years ago

@jaraco Please create a new distutils branch:

Ref #1191.

jaraco commented 6 years ago

Distutils branch now exists, created against the latest master. Is that what you had in mind?

ghost commented 6 years ago

Yes.

ghost commented 6 years ago

Is there anything left to do other than merging distutils to master?

ghost commented 6 years ago

Theoretically more testing is needed though.

ncoghlan commented 6 years ago

@xoviat @jaraco Is setuptools going to be adopting the still relevant parts of the legacy distutils documentation as part of this change?

We've kept https://docs.python.org/3/distutils/ and https://docs.python.org/3/install/ around and linked from https://docs.python.org/3/library/distutils.html because there are things documented in there that aren't documented anywhere else, but it would be nice to see that content migrated somewhere that's more actively maintained.

The other key question I would have relates to the setuptools injection in pip and other tools: existing versions of pip assume that simply importing setuptools into a process will be enough to migrate a "plain distutils" setup.py script to using setuptools instead.

While I know I suggested making that override opt-in in the original BitBucket discussion, I'm now thinking that you're going to have to keeping doing it implicitly as a side effect of import to maintain backwards compatibility with the expectations of installation tools.

ghost commented 6 years ago

So the current plan is to always attempt to override the builtin distutils package with the version provided by setuptools if setuptools is installed. In addition, importing setuptools should guarantee this behavior. Related to this, setuptools will preserve compatibility will all existing projects (numpy distutils included) as it has always done. Setuptools is not currently providing any documentation for distutils, but I see no reason why that would not be moved over (one step at a time).

ncoghlan commented 6 years ago

Cool, sounds like a good plan to me.

Given that, I think the one specific thing I would request is to have a dedicated page somewhere in the setuptools documentation about "Overriding distutils with setuptools". The premise behind that is to have a specific thing we can link to and/or update as folks find potential migration issues (similar to what we have for the pypi.org migration in https://packaging.python.org/guides/migrating-to-pypi-org/ ).

benoit-pierre commented 6 years ago

Looking at the code, setuptools will now patch it's own internal version of distutils, how will this work with pip? Pip expects setuptools to patch distutils so pure distutils projects are supported.

ghost commented 6 years ago

pip should continue to import setuptools to guarantee that distutils is patched. The monkeypatches will be migrated to to the internal distutils over time.

benoit-pierre commented 6 years ago

Right, provided setuptools is imported first, its version of distutils will be used.

jaraco commented 6 years ago

@seawolf42 @Zelgius FYI

jaraco commented 6 years ago

Substantial progress has been made on this effort in the distutils branch, though I haven't reviewed it.

pganssle commented 6 years ago

Looking at the distutils branch, it seems that we're just vendoring distutils there.

My thinking on the matter is that we should actually unify the codebases and just move the code from distutils directly into setuptools, rather than attempting to maintain distutils as a package within setuptools.

Basically I think the pypa/packaging-problems#127 model of making setuptools the reference implementation for the distutils API is better than maintaining an updated version of the distutils code. I think this is a good idea regardless of whether CPython uses an ensurepip style model to adopt setuptools as the reference implementation.

RonnyPfannschmidt commented 6 years ago

@pganssle i beleive in part that is entirely manpower dependen - i recall that @jaraco currently is usually only able to spark very little volunteer time every now and then, and hes pretty much the only active contributor im aware of

pganssle commented 6 years ago

@RonnyPfannschmidt He is not the only one working on setuptools, but in any case it's important to get the right decision.

In fact, I think it would be much worse if distutils moved into setuptools and there was only one person who doesn't have much time to work on it maintaining the project. CPython has a long release cadence but also has many contributors.

Most of the setuptools API is already subclassing and/or patching the base distutils classes as it is. I think we can start moving over those classes in their entirety rather than subclassing, and implement the patches directly in setuptools.

The main problems I could foresee this causing are people doing isinstance checks against distutils and people possibly accidentally relying on the setuptools monkey patches. Both of these may be fixable (at least to the degree that they are fixed by the "move distutils into the setuptools repo" solution) with a deprecation period.

pganssle commented 6 years ago

It's also worth noting that we can almost certainly do the "move all of the distutils components into setuptools" incrementally. Start with distutils.Distribution, for example, then Command, etc. Moving the entire package means that it all has to be done at once.

jaraco commented 6 years ago

I agree we don’t want setuptools to present distutils (which that Branch currently does). Rather, for the sake of incrementalism, I’d suggest that it include distutils as a private package (e.g. setuptools._distutils)... and eventually integrate the functionality into one unified implementation.

Perhaps it makes sense to have an even less ambitious adoption of only a portion of distutils.

I welcome any efforts, but compatibility (including the current expectation that some packages rely on the monkey patching of distutils) needs to be addressed, probably by deprecating distutils.

ncoghlan commented 6 years ago

While CPython as a whole has many contributors, we don't have many folks contributing to distutils any more - folks need build tools that let them target currently deployed versions of Python, which setuptools provides, but distutils really doesn't.

So the challenge for pypa/packaging-problems#127 is as follows:

The scope of the eventual distutils-compat shim probably wouldn't need to be the full distutils API, but it wouldn't be the empty set either.

So @jaraco's suggestion sounds like a reasonable starting point to me: give setuptools its own pre-patched copy of distutils so it can work on a distutils-free copy of Python, without providing an importable distutils facade.

Building a suitable facade (with a setuptools dependency) would then be part of removing distutils from the standard library.

pganssle commented 6 years ago

As an aside about implementing this, I could be way over-estimating the abilities of the average student, but this feel like approximately the right size project for a GSoC student. Too late for this year I'm sure, but if this isn't done by next year we might want to consider applying.

RonnyPfannschmidt commented 6 years ago

@pganssle while you consider the abilities of a student right, i believe there is an underestimation on the complexity of legacy not so nice apis and their interaction in setuptools - unfortunately both codebases are already hard to navigate, and their interaction makes things even more tricky

pradyunsg commented 4 years ago

unfortunately both codebases are already hard to navigate

As a past GSoC student, the thing that hits hard "right in the face" is the technical debt -- working with and around it, is very much not something that formal education prepares students for.

Anyway, now would be a good time to get started on figuring out GSoC-sized projects within the PyPA and identifying mentors for them, if we're interested in having projects under PSF's GSoC umbrella for 2020 with decent amount of preparation/planning.

jaraco commented 4 years ago

In #2143, I've drafted a PR that adopts Setuptools from CPython master, retaining history and attribution. This is essentially the second step outlined in pypa/packaging-problems#127. I've asked a few key contributors to review, but I welcome review or comments or questions from anyone.

hugovk commented 4 years ago

This is essentially the second step outlined in pypa/packaging-problems#217.

Typo, it's really: https://github.com/pypa/packaging-problems/issues/127 :)

jaraco commented 3 years ago

The Setuptools 48 release incorporates the fix and closes this issue, although there is still a lot of work to do to actually merge the functionality and remove the monkey patching.

77FN7 commented 1 year ago

tidelift.com/funding/github/pypi/setuptools