TLSv1.0/TLSv1.1 rolling brownouts are not transparent

[awallace-cray]

FYI, The pip clients we're using sure don't pass along any error 403 message. The connection "succeeds" but the client just won't find anything it is looking for (pip install). No response needed. Thanks.

[alex]

What version of pip are you using?

[awallace-cray]

Looks like 6.0.6. It is baked-in.

[alex]

Baked into what?

Can you try with the latest pip, 9.0.3?

[awallace-cray]

Baked-in the project config files. Which are probably up for grabs now anyway, because the TLSv1.0/TLSv1.1 change. Will let you know about trying pip 9.0.3. I think root cause on our side, is using old SLES-11 machines in this project.

[dstufft]

Ancient versions of OpenSSL will definitely be an issue.

[alex]

I don't know anything about SLES, but https://www.suse.com/documentation/suse-best-practices/singlehtml/securitymodule/securitymodule.html seems like the relevant doc

[awallace-cray]

Roger that. Will need new bits. Will take time. Meanwhile, a local mirror or cache or such, to keep project going. Tearing hair out today.

[ncoghlan]

Using devpi-on-recent-Python as a caching proxy could be a pretty robust workaround for older pip clients. I'm not sure we could provide that as a generic pre-assembled solution (since the specifics will differ depending on the exact constraints preventing a pip upgrade), but I believe it would let devpi handle the TLS v1.2 link to PyPI, while talking whatever the legacy client needs on the local side of things (for a local host loopback connection, it could even be HTTP, without using TLS at all).

[awallace-cray]

Our local PyPI mirror supports http and is working well. I did not build this mirror, and do not know how it works.

Pip install needs parameters --index-url $mirror_url --trusted-host $mirror_hostname, which can be passed to pip in different ways. All good.

However, many of our scripts also use easy_install. To go get virtualenv, for example. Easy_install works w our mirror if you add -i $mirror_url to its command line. I wish I had an "easier" way to pass easy_install parameters, like with conf files or an env variables. Ideally, this should work even with old versions of easy_install.

[alex]

We strongly recommend you stop using easy_install; it's been deprecated for a long time and hasn't really kept up with improvements to the packaging ecosystem.

[awallace-cray]

Good advice.

Do you happen to know when the TLSv1.0/TLSv1.1 policy will be fully enforced on pypi.org?

• Thanks

[alex]

We'll be gradually ratcheting up the brownout periods over the next few weeks; April 8th is our current target for fully disabling them.

[awallace-cray]

Said I'd get back to you on the pip error message.

time: 2018-03-23 08:06 PT host: an old SLES-11 machine, but I am using a recently-built Python 2.7 (user-built from source)

$ pip install argparse # No 403 error message
Collecting argparse
  Could not find a version that satisfies the requirement argparse (from versions: )
No matching distribution found for argparse

$ pip install -v argparse # there it is
Collecting argparse
  1 location(s) to search for versions of argparse:
  * https://pypi.python.org/simple/argparse/
  Getting page https://pypi.python.org/simple/argparse/
  Looking up "https://pypi.python.org/simple/argparse/" in the cache
  No cache entry available
  Starting new HTTPS connection (1): pypi.python.org
  "GET /simple/argparse/ HTTP/1.1" 403 109
  Status code 403 not in [200, 203, 300, 301]
  Could not fetch URL https://pypi.python.org/simple/argparse/: 403 Client Error: Brownout of Legacy TLS for url: https://pypi.python.org/simple/argparse/ - skipping
  Could not find a version that satisfies the requirement argparse (from versions: )
Cleaning up...
No matching distribution found for argparse

$ pip --version
pip 9.0.1 from /var/tmp/chapelu/awallace/venv/lib/python2.7/site-packages (python 2.7)

$ python --version
Python 2.7.13

$ type python
python is hashed (/ptmp/chapelu/awallace/venv/bin/python)

[alex]

@dstufft this seems like an issue, pip should, without -v, recognize a non-200 status code and print something useful. Is there an existing pip tracking bug for this?

[dstufft]

It did print the status code out right?

Could not fetch URL https://pypi.python.org/simple/argparse/: 403 Client Error: Brownout of Legacy TLS for url: https://pypi.python.org/simple/argparse/ - skipping

[dstufft]

I don't tihnk there's a bug for tracking any additional changes to that though, no.

[alex]

It only printed that with -v though -- it should do more by default.

[dstufft]

Oh right. Yea open an issue

[alex]

https://github.com/pypa/pip/issues/5111

[jvanasco]

FYI, pip 9.0.1 does not pass on any message or note a 403. It just says no matching versions are found, for everything. One must pass in -vvvv to see what happened.

[ntwaddell]

So what needs to be done to fix this? it's causing some of my python projects to fail in TravisCI

Could not fetch URL https://pypi.python.org/simple/pip/: 403 Client Error: Brownout of Legacy TLS

Just upgrade OpenSSL?

[jvanasco]

@ntwaddell try to upgrade pip first. you can download/run the get-pip.py from here: https://bootstrap.pypa.io/

that should fix several environments.

[alex]

What platform are you hitting this on with Travis CI? We should make sure any builtin packages they have are updated.

[ntwaddell]

@alex I'm using TravisCI Enterprise, it always takes them longer to update their enterprise platform.

Build language: python Build group: stable Build dist: precise Description: Ubuntu 12.04.5 LTS Release: 12.04

OpenSSL is OpenSSL 1.0.1 14 Mar 2012

[alex]

@ntwaddell On Ubuntu Precise you should be able to get a working OpenSSL by doing sudo apt-get update && sudo apt-get install --only-upgrade openssl libssl-dev.

[jvanasco]

@ntwaddell FYI, that's the first heartbleed release. That should have been upgraded years ago.

[brainwane]

@ntwaddell you should upgrade pip. Official release notes: https://pip.pypa.io/en/stable/news/

@awallace-cray asked:

Do you happen to know when the TLSv1.0/TLSv1.1 policy will be fully enforced on pypi.org?

The short answer, as Alex mentioned, is April 8th. More details:

As PSF blogged last year https://pyfound.blogspot.com/2017/01/time-to-upgrade-your-python-tls-v12.html , on June 30, 2018, all Python.org sites are going to entirely stop supporting TLS versions 1.0 and 1.1, because our CDN provider is deprecating support for those versions.

We are launching the new PyPI (in beta at https://pypi.org) this month and replacing the legacy PyPI (https://pypi.python.org). Here's the beta announcement for the new PyPI: https://pyfound.blogspot.com/2018/03/warehouse-all-new-pypi-is-now-in-beta.html Warehouse, the codebase for the new PyPI, does not support TLS 1.0 or 1.1.

So, as of late March, the Python Package Index has started doing rolling brownouts of the deprecated TLS versions. For some portion of each hour, anyone attempting to access PyPI with TLSv1.0 or TLSv1.1 is supposed to get a 403 response with an informative error. As we get closer to the deadline, we will be ramping up the amount of time the endpoint is down for the deprecated TLS versions. The ultimate goal is to have it be 100% unavailable on and after April 8th, prior to the final deadline (because we can give a good error message -- once June 30th hits, it will just be an uninformative OpenSSL error).

More info:

• https://github.com/pypa/warehouse/issues/3293
• https://github.com/pypa/warehouse/issues/3411
• https://status.python.org/incidents/btjtz01lzp88

[jvanasco]

@brainwane said:

The ultimate goal is to have it be 100% unavailable on and after April 8th, prior to the final deadline (because we can give a good error message -- once June 30th hits, it will just be an uninformative OpenSSL error).

Have you looked into doing a rolling brownout that triggers OpenSSL errors after April 8th?

The informative 403 response is entirely swallowed by some pip versions. An OpenSsl error might raise an uncaught exception, and lead people into checkout out the website (where they'd find info on the migration).

[alex]

I'm not sure we have the technical ability to trigger the OpenSSL error in advance of our CDN -- it requires changing how they perform the TLS handshake.

[jvanasco]

I totally forgot there's a cdn :(

[dstufft]

We can trigger tls errrors pretty much whenever we want, but it’s all or nothing and a one way switch. Once we disable it, it’s gone.

Sent from my iPhone

On Apr 2, 2018, at 2:45 PM, Jonathan Vanasco notifications@github.com wrote:

I totally forgot there's a cdn :(

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[tueksta]

Brownout by sending 403 for "some portion of each hour" is a horrible decision. We have flickering CI due to this and with the logs not showing the actual error unless you escalate the verbosity levels it leaves everyone wondering why something that has been working perfectly until now sometimes fails and sometimes succeeds.

[jjshoe]

Fix for osx?

[alex]

@jjshoe Upgrading to the latest version of pip (9.0.3) should always resolve this issue on macOS. If pip install -U pip doesn't work because of the brownout, curl https://bootstrap.pypa.io/get-pip.py | python - will.

[shrielastica]

@alex: Even after pip upgrade, still the issue is seen. Please advise

04-05-2018 16:43:32 INFORMATION - [Execute ID: 12] : Staring execution of command line: /Library/Frameworks/Python.framework/Versions/3.5/bin/pip3 --verbose --trusted-host=pypi.python.org install requests 04-05-2018 16:43:32 INFORMATION - [Execute ID: 12] : BEGIN COMMAND OUTPUT 04-05-2018 16:43:33 INFORMATION - [Execute ID: 12] : The directory '/Users/automation/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. The directory '/Users/automation/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. Collecting requests 1 location(s) to search for versions of requests:

• https://pypi.python.org/simple/requests/ Getting page https://pypi.python.org/simple/requests/ Starting new HTTPS connection (1): pypi.python.org "GET /simple/requests/ HTTP/1.1" 403 109 Could not fetch URL https://pypi.python.org/simple/requests/: 403 Client Error: Brownout of Legacy TLS for url: https://pypi.python.org/simple/requests/ - skipping Could not find a version that satisfies the requirement requests (from versions: ) Cleaning up... No matching distribution found for requests Exception information: Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main status = self.run(options, args) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/commands/install.py", line 324, in run requirement_set.prepare_files(finder) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_set.py", line 380, in prepare_files ignore_dependencies=self.ignore_dependencies)) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_set.py", line 554, in _prepare_file require_hashes File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/req/req_install.py", line 278, in populate_link self.link = finder.find_requirement(self, upgrade) File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/index.py", line 514, in find_requirement 'No matching distribution found for %s' % req pip.exceptions.DistributionNotFound: No matching distribution found for requests Starting new HTTPS connection (1): pypi.python.org "GET /pypi/pip/json HTTP/1.1" 403 109 There was an error checking the latest version of pip Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/utils/outdated.py", line 128, in pip_version_check resp.raise_for_status() File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pip/_vendor/requests/models.py", line 862, in raise_for_status raise HTTPError(http_error_msg, response=self) pip._vendor.requests.exceptions.HTTPError: 403 Client Error: Brownout of Legacy TLS for url: https://pypi.python.org/pypi/pip/json

[alex]

@shrielastica what OS version is this?

[gajjar8055]

I have spent around 5 hours and almost die, man

[shawngustaw]

I'm not sure if this is related at all but a coworker was having similar issues and brew install python@2 ended up fixing it. Oddly, downloading Python 2.7.14 directly from the Python website didn't work, but installing via brew and overwriting seems to be using the correct OpenSSL version.

[awallace-cray]

I can confirm: brew install python@2 installs newer version of easy_install and pip in /usr/local/bin, and these clients can connect to PyPI even when the system-installed Python-2 easy_install or pip cannot.

I've read posts about brew install python fixing the same or similar problem for Python 3.

[brainwane]

A comment on LWN suggests we should have an announcement about this on python.org. Someone could write something to post to http://blog.python.org/ which syndicates to that front page -- caution that only about the first ~30 characters will show up.

[alex]

At this point we've had TLS 1.0/1.1 disabled 100% of the time for a few days with limited feedback, I'm reluctant to think we need further publication of this.

[damienrj]

I know this caught my company by surprise and has broken a few things.

[alex]

Can you suggest what communications channels we could have promoted the brownouts on to help you see them?

[brainwane]

Now that we have entirely stopped supporting TLS 1.0 and 1.1, I am closing this issue. To be notified of future breaking changes to PyPI, please subscribe to the pypi-announce mailing list.