requests 2.14.0 cannot be installed using pip < 8.1.2

[lmazuel]

Example with pip 6.1.1 (but same with pip 8.1.1):

> pip install requests
You are using pip version 6.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting requests
  Using cached requests-2.14.0-py2.py3-none-any.whl
  Exception:
  Traceback (most recent call last):
    File "D:\VEnvs\testpip\lib\site-packages\pip\basecommand.py", line 246, in main
      status = self.run(options, args)
    File "D:\VEnvs\testpip\lib\site-packages\pip\commands\install.py", line 342, in run
      requirement_set.prepare_files(finder)
    File "D:\VEnvs\testpip\lib\site-packages\pip\req\req_set.py", line 345, in prepare_files
      functools.partial(self._prepare_file, finder))
    File "D:\VEnvs\testpip\lib\site-packages\pip\req\req_set.py", line 290, in _walk_req_to_install
      more_reqs = handler(req_to_install)
    File "D:\VEnvs\testpip\lib\site-packages\pip\req\req_set.py", line 557, in _prepare_file
      set(req_to_install.extras) - set(dist.extras)
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2758, in extras
      return [dep for dep in self._dep_map if dep]
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2781, in _dep_map
      self.__dep_map = self._compute_dependencies()
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2814, in _compute_dependencies
      common = frozenset(reqs_for_extra(None))
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2811, in reqs_for_extra
      if req.marker_fn(override={'extra':extra}):
    File "D:\VEnvs\testpip\lib\site-packages\pip\_vendor\_markerlib\markers.py", line 113, in marker_fn
      return eval(compiled_marker, environment)
    File "<environment marker>", line 1, in <module>
  NameError: name 'platform_system' is not defined

So, installing requests 2.14.0 implicitly requires pip >= 9.x. Should be at least in the release note as a disclaimer, or be fixed if it's not on purpose.

[tgamblin]

@SethMichaelLarson: is there an easy way to check for this type of thing in setup.py? I'm assuming it gets control at some point or another.

[Lukasa]

@tgamblin Not if you install the wheel, which the vast majority of our users do. Wheels run zero package code on install.

[adamjstewart]

@tgamblin The problem is that setup.py never gets executed. pip downloads wheels.

[sethmlarson]

@Lukasa Might be a good time to have that conversation. This project has a unique position in being kind of the first in the line of a few major packages that can stir change in the ecosystem, might be a good idea to have the answers to these types of problems formalized.

[cjw296]

It's certainly saddening to see "base" packages like pip and requests are taking more of an approach of "well, we're using the latest cut of master, so everyone else needs to now too". I guess lack of stability and endless work for library maintainers who try and do proper CI are the way of the future in the Python ecosystem?

[Lukasa]

Per @nsoranzo's link it looks like we have some reason to assume that sys_platform might be better supported, potentially adding another year of pip/setuptools releases. I presume from mitmproxy using it that this does not cause problems on Travis-CI, which tackles a large chunk of the issue. If that's true, and Travis-CI is using a 6.x release series pip, then we probably gain back 10% of the install base, which is a fairly compelling option.

[kylape]

So okay I need to upgrade setuptools as well on RHEL 7 to both run our build and run sphinx-build. pip install --upgrade pip setuptools produces this output:

  Running setup.py egg_info for package setuptools
    Traceback (most recent call last):
      File "<string>", line 3, in <module>
      File "setuptools/__init__.py", line 10, in <module>
        from six.moves import filter, map
    ImportError: No module named six.moves
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 3, in <module>

  File "setuptools/__init__.py", line 10, in <module>

    from six.moves import filter, map

ImportError: No module named six.moves

So now I have to upgrade six, setuptools, and pip, like so:

pip install --upgrade pip six
pip install --upgrade setuptools

[Lukasa]

@cjw296 I'm sorry you think that's what we're doing. I think this issue represents a reasoned and reasonable discussion of the pros and cons of this decision, with no definitive position being taken yet, but that's just me.

[Lukasa]

For people interested in a possible mitigation, see #4007.

[Lukasa]

BTW, if anyone has a system that has an old pip on it, feel free to try pip installing from my github branch in #4007 to see if that resolves the issue for you. That data would be helpful.

[alex]

I don't want to be dismissive about concerns, as this did break some things, but if you're hitting this, it's because you're running the very latest version of requests, released like an hour ago. It doesn't seem totally out of line to say that if you're running the very newest version of a package, you might also need a new version of something else.

The desire for a "stable" OS (e.g. RHEL, CentOS, Ubuntu LTS), with all the latest packages and all the oldest tools is a contradiction in terms; and pretending like this is a reasonable practice is a massive maintenance burden for OSS developers.

[dstufft]

if you're hitting this, it's because you're running the very latest version of pip, released like an hour ago

You mean requests right?

[alex]

Yes, sorry. Editing.

[sethmlarson]

@alex As you can probably tell from my comments, I am in complete agreement.

[martin-langhoff]

@alex the real world is a lot more complex than that. As I stated earlier in this discussion thread, requests is breaking things for folks who don't depend on requests! I quote my earlier msg:

Note that lots of folks don't depend on requests; we don't! But we pip install a package that through its depchain does. Our needs re stability tradeoffs might not align with the needs of every maintainer in the depchain.

[martin-langhoff]

I've been author/code/maintainer of various things you all probably use. Pretty avatars hide my grey hairs... all I'll say is: When a lot of people pile up in a thread saying "this update broke me", it's a good idea to figure out how to unbreak things... which I am glad seems to be happening.

I didn't know I was using requests, the requests maintainers didn't know so many people were using requests, in some cases indirectly. Sobering, right? It's a privilege. It's a responsibility. You're part of infra others use.

[sethmlarson]

@martin-langhoff Requests is #1 downloaded from PyPI. I'm pretty sure maintainers knew that people used Requests quite a bit. :)

[cjw296]

Right, so the source of pain here generally seems to be Sphinx, which is what depends on requests and setuptools. So, currently, to safely build on CI, you now need to do:

pip install --upgrade pip six
pip install --upgrade setuptools
...now what you actually wanted to do.

I wonder how many more of these will need to be added before library maintainers just give up?

[sethmlarson]

@cjw296 That's a one-time two-line change to .travis.yml until Travis upgrades their default packages.

[edmorley]

For end-users whose projects have been surprise-broken by this change - I would strongly recommend pinning your dependencies via --require-hashes mode (which warns about unpinned sub-dependencies amongst other things): https://pip.pypa.io/en/stable/user_guide/#ensuring-repeatability https://pip.pypa.io/en/stable/reference/pip_install/#hash-checking-mode

Doing so used to add hassle in the form of having to continually update requirements files by hand, but there are now tools out that support automatically updating the pinned versions even when they have hashes (eg https://pyup.io).

For a project I work on there was no surprise breakage, just a bot-opened PR with a failing Travis run.

(Alternatively check out https://github.com/kennethreitz/pipenv if you don't need bot support.)

[Lukasa]

@martin-langhoff We were totally aware of how many people used Requests. The problem was that we were not aware of how many of those users were using older infrastructure to install current versions of Requests, such that our attempt to fix things for Windows users would lead to breakage on those systems. Generally speaking this has not been data we've monitored, because it's not been enormously useful historically, but presumably we'll need to change that in the future.

[cjw296]

@SethMichaelLarson - yeah, so was the last one (which was pip vs six). I'm pretty sure there was one before that too.

@edmorley - the problem is that, as a responsible library developer, I want to try and use the latest stable releases of packages so I don't create dependency hell problems for users of those libraries.

Sure, I can put a requests < 2.0 in my library's setup.py, but is that going to help someone who wants to use my library bute requires a feature of requests in their own code that's only in 2.0+?

[sigmavirus24]

So I wasn't around for the change in question, but reading the thread there are a few things worth noting:

1. Some people are trying to be helpful but antagonizing others who are experiencing pain and those people need to stop.

2. This change was made as a best effort approach to unbreak users without breaking far more users. It had the unintended consequence of educating people about how ancient their versions of pip, setuptools, etc. are.

3. The error here fundamentally stems from old versions of critical infrastructure. Yes, @martin-langhoff, some of our consumers are unknowingly consuming requests through other packages that expect people to be using reasonably recent (not the latest) versions of software. If anything, this further highlights the fact that most people pay little attention to their dependencies.

4. Making a data driven decision for this issue is hard with people repeating themselves trying to be correct. There is no one right answer. There is just a least bad option in this case. I'd encourage people to take a walk and get away from their keyboards. Tensions are unnecessarily high. There are several workarounds and #4007 is a potential permanent fix.

[Balooga]

NameError: name 'platform_system' is not defined is default Sphinx project.

I just tried make html in a default sphinx project in both OSX and CentOS7.

Both have the latest version of PIP installed.

pip --version pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)

On OSX, the default project compiles to html successfully.

On CentOS7, I receive the following error;

Exception occurred: File "<environment marker>", line 1, in <module> NameError: name 'platform_system' is not defined

[Lukasa]

Whoops, this shouldn't have been auto-closed.

[Lukasa]

@Balooga Try updating setuptools too.

[Lukasa]

Ok folks I just shipped v2.14.1, which should vastly widen the number of supported pip and setuptools versions. Please all try using the new release and see how it works for you.

[Balooga]

@Lukasa

sudo pip install -U setuptools works. Thank you.

[martin-langhoff]

@Lukasa - bravo thank you! Works on CentOS6 with python2.7 from SCL, pip 7.1.0 .

[mDuo13]

The updated version un-broke our CircleCI builds (which are locked to Ubuntu Trusty or older). Thanks!

[devops-dude]

I'm so grateful you decided to fix this as it broke at least two of our Ubuntu based stacks in QA and in Integration. Thank god we deployed production early this morning! People read instructions to install things like New Relic or Rollbar which use pip. It really isn't the end users fault that Ubuntu 16.04 LTS is using an older version of pip and breaks. I doubt many people installing tools like New Relic have a clue what "requests" is.

Ubuntu 14.04 - pip 1.5.4 Ubuntu 16.04 - pip 8.1.1 (That's why those two versions are 2.1 million downloads in the past 30 days)

People lock in versions thinking the release isn't going to change. When it goes and grabs updating packages it can come as quite a surprise.

Over the 4 hours this was broken it took two engineers about 3 hours to identify the problem and start fixing servers. Another 30 QA+Dev engineers were unable to deploy and test changes. I can't imagine how many people were impacted as a whole. Short term I'll be up tonight adding upgrading pip to the latest version in all of our pipelines. Long term I will try to make sure we always use our package manager and never use pip to install anything on a server.

[tgamblin]

Thanks to @Lukasa for handling this so well. It's not always easy to deal with the users, especially when they're all on your GitHub issue at once, but it's clear the requests folks care 😄 .

[ncoghlan]

Well done on finding a mitigation that improves compatibility with earlier pip versions.

Longer term, it likely makes sense to use this case as an example for why wrapping a virtual environment in an immutable deployment artifact (e.g. container, deb, RPM) or locking all your transitive Python level dependencies (e.g. with pip-tools or Pipfile.lock) is a good idea - even the most diligent of developers can accidentally introduce incompatibilities with older distros, and tools like pipenv and pyup.io help ensure your worst case scenario in such instances is a failure in pre-merge CI for a dependency update rather than a broken deployment pipeline or production environment.

[Lukasa]

@tgamblin Thanks to all of you! We've had stuff like this in the past, and it's really easy for a GitHub thread like this one to get heated and out of control. It's always good to have a community work together, both maintainers and users, to try to find some kind of reasonable outcome that helps everybody. So far I think we've done that, though we'll keep a careful eye on bug reports for the next few weeks and months to see whether we've got an ongoing long-tail of users who are encountering trouble.

@ncoghlan Yeah, there are still some wider Python community things we can look to do to try to reduce the pain of these updates. The pip maintainers have made it clear to me that in general pip is upgraded very frequently: after a release it takes less than 24 hours for the plurality of downloads to be coming from the newest pip release. Looking at our download graphs, the older pips with large download counts can almost always be traced back to the system pip shipped with Linux distributions.

This suggests that there are users that have landed on a developer workflow that involves using the system Python toolchain without appropriate bounding on versions. As has been pointed out by upthread by @martin-langhoff this is in many cases happening transitively, where the top-level dependency is pinned but it has an unpinned dependency in its dependency tree, but it nonetheless leads to a place where users are hoping to build a stable environment and not getting one. It's one thing when the breakage happens due to projects being insufficiently careful with their versioning, it's quite another when a project becomes incompatible with the toolchain you're using!

Hopefully we'll find a way to help developers in situations like these use better tooling to ensure that the stable environment they want to have is the one that they actually do have.

---

And with that, I'm going to close this thread. I had a whole night's sleep without anything going particularly wrong with our mitigation, which suggests it's working pretty well for now. Of course, if people run into trouble with it you should feel free to comment here and we'll work with you to try to resolve it. Thanks for all your time everyone! :heart: :cake: :sparkles:

[origamee]

Folks, I upgraded requests using sudo pip install --upgrade requests and now I can't import requests any more. It says:

ImportError: No module named requests

So I've removed the requests module multiple times and re-installed, but nothing works. I'm using python2.7 on Centos release 6.9

[Lukasa]

@ssmasud You must not sudo pip install --upgrade requests on a Linux machine. Centos already installs Requests as a system package, and doing a sudo pip install --upgrade requests will overwrite that system package with a newer version that is not compatible with the other packages you install.

On Linux, if you are pip installing Requests to get a new version you must use a virtualenv.

[origamee]

ok @Lukasa so how do I make the requests module functional again?

[origamee]

or in short, how do i repair python2.7 installation on centos?

[ncoghlan]

@ssmasud This is probably a more appropriate question for Ask Fedora or Server Fault, but the key step is to ask yum to restore the system version:

sudo yum reinstall -y python-requests

If that doesn't fix the problem, you may also need to reinstall the system versions of request's dependencies (it's probably a good idea to do that anyway, just to be on the safe side):

python-backports
python-backports-ssl_match_hostname
python-six
python-urllib3

[rfleschenberg]

I am not very familiar with centos, but it might be a good idea to have a look at your /usr/local/lib/python* tree, or check where pip installed requests to and clean that up.

[origamee]

Thanks guys. Looks like pip2.7 install <whatever> did the trick