Drop support for self upgrade/installation

[jaraco]

I'd like to consider dropping support for setuptools installing or upgrading itself, including the removal of the bootstrap script, and instead, rely entirely on pip for installation of setuptools. Doing so would (a) allow for the simple declaration of dependencies over vendoring them, (b) eliminate the need for a bootstrap process besides simple 'pip install', (c) eliminate the special handling for self installation, and (d) limit setuptools installation to the single-version installs that pip allows.

I expect some environments like bootstrap or other systems might be affected, so I'm cautious about this possibility. This ticket serves to host the conversation and considerations.

[bhaugen]

I'd suggest you also put some official upgrade instructions in the readme and docs. I don't see anything there, and I see the usual conflicting advice on the internet. Same for pypi.

I tried this, which I found in some comments here: sudo pip install setuptools –upgrade –no-use-wheel which seemed to work fine.

[jaraco]

The no-use-wheel recommendation is concerning, because that option indicates to setuptools that it needs to build itself, which is contrary to the goal of this ticket. I think one assumption is that setuptools would always be installed from wheels, and the build step would only be invoked on systems that already had a setuptools installation.

But you're right, the upgrade story needs to be well understood and documented.

[bhaugen]

I hope it's clear that I was not recommending –no-use-wheel, just copying and pasting.

[jayvdb]

I have recently needed to install setuptools manually because pip couldn't do it.

https://github.com/PyCQA/pyflakes/pull/76/files#diff-11c909939117928998b102a1fff7d363R33

[pombredanne]

I think it makes a lot of sense for a cleaner and consistent experience and clean separation of duties. It will make the overall package landscape more coherent: setuptools builds (and optionally installs from sources) eventually programmatically while pip bootstraps and installs.

The only area where I can fathom a possible issue for some rare tools is the lack of a stable function API in pip when using it as a library. But even then, spawning a pip subprocess is not a big issue (instead of calling its functions) from Python and this is something I routinely do.

[sigmavirus24]

@jayvdb let's be entirely forthright here, pyflakes needed to install setuptools manually on AppVeyor and pypy. That also should have been a separate bug report to fix that. It's related to this, yes, but not something that negates this goal.

[jaraco]

In the feature/581-depend-not-bundle branch, I've drafted an implementation that passes tests and installs nicely as a wheel. If you would like to try it out, for a limited time, the wheel of g767dcea0 can be installed as so:

pip install https://dl.dropboxusercontent.com/s/nn2nh33fvqkz2u5/setuptools-33.0.0.post20170101-py2.py3-none-any.whl

[AraHaan]

hmm I think having it upgrade itself (as an optional configuration option) should be added and default to not set to do it. So that way those who prefers this self upgrades can do it but before it upgrades itself it checks for upgrades to the other dependencies instead of shipping them with setuptools. But wouldn't this break pip for a little while as some parts of pip use setuptools?

So basically (as an optional config) have to self updates go in this order:

1. Any updated dependencies that setuptools requires? install them.
2. Any updates for setuptools? if so install them.

This way for those who (like me) does not always remember to pip install modules to know when there is a new version it would be easier to handle. Besides I have been working on things like this for my code so I dont have to remember to update them constantly to robot pip.

[jaraco]

@AraHaan I don't think there's any way to achieve what you describe. Part of the problem is that easy_install of any package over a pip-installed version of that package is liable to break. easy_install doesn't know how to uninstall pip-installed packages, so I wouldn't even expect a self-upgrade to work.

Now if you're talking about as strictly easy_install-managed environment, perhaps your suggestion would be plausible, but still very tricky. The way easy_install works is it downloads the sdist of the target package (i.e. setuptools), expands it to a temporary folder, builds a bdist_egg of that package, installs that egg, then determines which unsatisfied dependencies exist and installs them. The problem lies in the 'build' step, which because a 'setuptools' directory is present in that build directory, means that it's already using the newer version of setuptools to build itself. One would have to specifically special-case setuptools to inspect that package metadata and determine its dependencies to install those in advance. While possible, this approach would have additional pitfalls. What if this new version of setuptools relies on a dependency that's not compatible with the dependency that the current version of setuptools requires? Then this pre-install of dependencies might succeed but the install of the new version of setuptools might not succeed, leaving the environment in a broken state. Additionally, the way that setuptools isolates itself from the installation of the target package is to use in-process tricks (sandbox module) to hide existing imports of setuptools - that approach would also need to be expanded to include hiding of existing imports of dependencies.

In other words, it may be possible, but it's an awful lot of work and investment in a tool chain that's long been superseded by pip. If others wish to invest this effort, I won't resist it, but personally, my focus will be on pushing to the future rather than dragging forward the past.

[jaraco]

I've addressed the concern above about pyflakes on appveyor by providing a technique for getting pip installed and using that to install setuptools on pypy.

I'm ready now to proceed with the release. I've been running the branch version of setuptools for some time now with success. I'd like to invite the community one last time to give this version a try and identify issues that may arise.

Here's a link to the wheel: https://dl.dropboxusercontent.com/s/4wwpuzpg1ortmgn/setuptools-33.1.1.post20170116-py2.py3-none-any.whl Here's a link to the sdist (though you won't be able to pip install it without first satisfying dependencies): https://dl.dropboxusercontent.com/s/r1aae2zqik5px0h/setuptools-33.1.1.post20170116.zip

Please try it out and identify any issues in your environment before I cut a release.

[jaraco]

Oh, and if you're using the ez_setup.py bootstrap script, please try the version found in the bootstrap-581 branch.

[dstufft]

This may be a silly idea, but it seems like with this branch it'd make sense to just deprecate and remove ez_setup.py completely. It's current incarnation feels like it's likely to be fairly fragile since it depends on having pip installed already (and one of the cases where someone might use ez_setup.py is if they have an empty environment). It seems like directing people to use get-pip.py (which will currently install pip, setuptools, and wheel) and doesn't depends on anything but Python existing is a better solution, but I may be missing something.

[jaraco]

Not a silly idea at all. My goal was to be minimally invasive and provide a compatibility shim that would ease the transition and guide its users to the same conclusion (just use pip directly). I was particularly concerned about the bootstrap scenario, which invokes ez_setup.py through its API. They bundle ez_setup.py in their own bootstrap script. But now that you say it, it's probably the case that their bootstrap will fail at the implicit assumption that pip is installed, in which case it's no help.

It seems to me the options are to provide an ez_setup.py script that tries to do the right thing or to break it down and have it do nothing but raise an error message that it's no longer supported. Leaving it in place as-is will only lead users to an error during the install, which will appear like a bug:

$ curl https://bootstrap.pypa.io/ez_setup.py -O
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12764  100 12764    0     0  53073      0 --:--:-- --:--:-- --:--:-- 53183
$ python -m venv --without-pip ~/.envs/ez-setup     
$ ~/.envs/ez-setup/bin/python ez_setup.py --version=33.1.1.post20170116 --download-base=https://dl.dropboxusercontent.com/s/r1aae2zqik5px0h/
Downloading https://dl.dropboxusercontent.com/s/r1aae2zqik5px0h/setuptools-33.1.1.post20170116.zip
Extracting in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpltp77_zq
Now working in /var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpltp77_zq/setuptools-33.1.1.post20170116
Installing Setuptools
Traceback (most recent call last):
  File "setup.py", line 11, in <module>
    import setuptools
  File "/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/tmpltp77_zq/setuptools-33.1.1.post20170116/setuptools/__init__.py", line 10, in <module>
    from six.moves import filter, map
ModuleNotFoundError: No module named 'six'
Something went wrong during the installation.
See the error message above.

I'll go with the latter option and simply deprecate ez_setup.py.

[dstufft]

Not so much a suggestion but really just a statement that this is going to currently make installing setuptools via sdist via pip more annoying since people will have to manually install setuptools' build dependencies (since pip has no concept of that yet, and setup_requires won't work in this case).

That being said, https://github.com/pypa/pip/pull/4144 should hopefully land in the near future and allow setuptools to add a pyproject.toml file to itself that looks like:

[build-system]
requires = [
    "wheel",
    "packaging>=16.8",
    "six>=1.10.0",
    "appdirs>=1.4.0",
]

and pip will basically do what rwt setup.py would do, except with just those dependencies.

[jaraco]

Yes, thanks for that clarification and reference. The expectation is that users will typically install from wheels and anyone who wants to install from sdist will need to prep the environment in advance. Glad to hear that PEP 518 support is coming to pip.

[dstufft]

@jaraco I feel like this also paves the way for splitting pkg_resources out into its own package right? One that setuptools could depend on for both its own uses, and so everyne who currently used the setuptools == pkg_resources assumption in their dependency declaration is not broken, but start to allow people to depend on pkg_resources itself instead of all of setuptools?

[jaraco]

As you can see ^ I've updated the bootstrap script to simply pin to the last version prior this new release, and that compatibility bootstrap script, while warning about the issue, should continue to support older environments.

[shufflebits]

Sigh. This breaks a fresh install of Centos 7 1) Install CentOS-7-x86_64-Minimal-1611.iso 2) run yum -y update 3) run yum -y install python-setuptools 4) easy_install pip

The last line now fails, with TypeError: call() takes exactly 2 arguments (1 given)

What is the approved method nowadays for installing pip? This failure came as a nasty surprise in an install script which has worked unchanged for a year or so.

[dstufft]

curl https://bootstrap.pypa.io/get-pip.py | python.

[shufflebits]

Hmm. Still embarassing. And the change to the behavoiur is not reported - I just got a load of mysterious errors. It; s taken an hour I won't get back, chasing this down. At the time when the unsupported operation took place, there was no message, or error.

[jaraco]

Some of the motivations that led me to filing this ticket originally:

1. It's better in principle for packages to declare their dependencies rather than vendor them.
2. pip already vendors its dependencies, and pip is the blessed package that has special handling for bootstrapping (through ensurepip, etc). Setuptools should be able to rely on that.
3. Updating vendored packages without making mistakes is much more difficult.
4. Vendoring made it impossibly complex to extract pkg_resources as a separate package (because both would need vendoring, and that causes more nesting).
5. Pull requests for code in vendored packages would have to be handled and rejected.
6. A fallback technique needed to be developed and maintained to allow system package managers to unvendor the dependencies.
7. The vendoring technique itself isn't a well-defined process, so itself developed over several iterations and was inconsistent with the vendoring technique pip uses.
8. Vendoring increases the risk and challenges involved with adopting new dependencies.
9. Vendoring masks the actual dependencies (nothing was declared in install_requires and equivalent metadata).
10. The PyPA seeks to make setuptools a tool for building, pkg_resources for metadata access, and for these to be treated mostly like normal packages.

There are probably others, but those are the ones that come to mind. The main downside that comes to mind, aside from the various edge cases that are being addressed, is that of bootstrapping from source, especially when the dependencies require setuptools to build from source, being discussed in #980.

[dgreis]

Hi there, I'm not a python expert at all, but the dockerfile I wrote in November broke when I just tried to rebuild the container today. Specifically, it choked on the line:

RUN wget https://bootstrap.pypa.io/ez_setup.py -O - | python

However, when I replaced it with:

RUN curl https://bootstrap.pypa.io/get-pip.py | python

everything returned to normal. This github thread is quite confusing to follow for python dilettantes like myself. I basically had to guess which line of code to borrow to fix it, and simply got lucky. ( it ended up being the line from the comment from @dstufft on 2/8; thanks for that! 😌 )

Anyway, @jaraco mentioned that there might be edge cases to address in the work you guys are doing. I'm not sure if mine is one of them, but it would be helpful to have some straightforward instruction somewhere that just tells simplistic users to substitute the one line for the other. Thanks and keep up the great work. I love using python!

[projekt01]

Great, you killed our production deployment again, The third time in the last 6 month.

[pombredanne]

@projekt01 what do you mean exactly? If you have encountered an issue it would be useful if you can report it and post actual details. Thanks!

[astrofrog]

Just to be clear, is the recommended route for packages previously requiring ez_setup.py to simply remove the file? Is the idea that if installing e.g from a cloned repo, one should then either do:

pip install .

or

pip install setuptools  # if not already installed
python setup.py install

?

[jaraco]

Yes, one should do pip install ..

[Mirona]

FYI, this completely messed up this tutorial and I don't know how to proceed: http://newcoder.io/begin/setup-your-machine/#windows

https://github.com/econchick/new-coder

How should the tutorial here be updated to reflect the changes? @econchik

[The-Compiler]

@Mirona see https://packaging.python.org/tutorials/installing-packages/

[Mirona]

Thank you @The-Compiler, was just looking there...

On this step: https://packaging.python.org/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date

There is an error:

Could not fetch URL https://pypi.python.org/simple/pip/: There was a problem confirming the ssl certificate: <urlopen error [Errno 1] _ssl.c:504: error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version> Will skip URL https://pypi.python.org/simple/pip/ when looking for download links for pip in c:\python27\lib\site-pac kages\pip-1.4.1-py2.7.egg