pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.51k stars 3.02k forks source link

Network failure leads to TypeError in retry #3282

Closed JohannesBuchner closed 8 years ago

JohannesBuchner commented 8 years ago

I got this traceback after a network failure with urllib3 v1.10 on Linux:

Exception:
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connection.py", line 135, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/util/connection.py", line 66, in create_connection
    for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
  File "/usr/lib64/python3.4/socket.py", line 533, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 559, in urlopen
    body=body, headers=headers)
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 782, in _validate_conn
    conn.connect()
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connection.py", line 215, in connect
    conn = self._new_conn()
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connection.py", line 144, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f86e18d8198>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/pip/basecommand.py", line 211, in main
    status = self.run(options, args)
  File "/usr/lib64/python3.4/site-packages/pip/commands/install.py", line 294, in run
    requirement_set.prepare_files(finder)
  File "/usr/lib64/python3.4/site-packages/pip/req/req_set.py", line 334, in prepare_files
    functools.partial(self._prepare_file, finder))
  File "/usr/lib64/python3.4/site-packages/pip/req/req_set.py", line 321, in _walk_req_to_install
    more_reqs = handler(req_to_install)
  File "/usr/lib64/python3.4/site-packages/pip/req/req_set.py", line 461, in _prepare_file
    req_to_install.populate_link(finder, self.upgrade)
  File "/usr/lib64/python3.4/site-packages/pip/req/req_install.py", line 250, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 486, in find_requirement
    all_versions = self._find_all_versions(req.name)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 404, in _find_all_versions
    index_locations = self._get_index_urls_locations(project_name)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 378, in _get_index_urls_locations
    page = self._get_page(main_index_url)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 818, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/lib64/python3.4/site-packages/pip/index.py", line 928, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 480, in get
    return self.request('GET', url, **kwargs)
  File "/usr/lib64/python3.4/site-packages/pip/download.py", line 373, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib64/python3.4/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib64/python3.4/site-packages/cachecontrol/adapter.py", line 46, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "/usr/lib64/python3.4/site-packages/requests/adapters.py", line 370, in send
    timeout=timeout
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/connectionpool.py", line 609, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib64/python3.4/site-packages/requests/packages/urllib3/util/retry.py", line 226, in increment
    total -= 1
TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'

Somehow pip has a Retry object which is incompatible. I reported this first with urllib3 in https://github.com/shazow/urllib3/issues/742 who say it is a bug in pip (or requests).

JohannesBuchner commented 8 years ago

I used pip3.4 from Gentoo repos.

I used pip to install a local egg directory, which loaded some dependencies off the web (sqlalchemy to be precise)

I did not trim the traceback.

xavfernandez commented 8 years ago

@JohannesBuchner: It looks like the pip dependencies have been debundled. It would be interesting to know what version of the dependencies you have compared to what pip 7.1.2 expects: https://github.com/pypa/pip/blob/7.1.2/pip/_vendor/vendor.txt

JohannesBuchner commented 8 years ago

I think Gentoo does not debundle

the following is from the ebuild file

Check pip/_vendor/vendor.txt for this

VENDOR_DEPEND="

=dev-python/distlib-0.2.1[${PYTHON_USEDEP}] =dev-python/html5lib-0.999999[${PYTHON_USEDEP}] =dev-python/six-1.9[${PYTHON_USEDEP}] =dev-python/colorama-0.3.3[${PYTHON_USEDEP}] =dev-python/requests-2.7.0[${PYTHON_USEDEP}] =dev-python/CacheControl-0.11.5[${PYTHON_USEDEP}] =dev-python/lockfile-0.10.2[${PYTHON_USEDEP}] =dev-python/progress-1.2[${PYTHON_USEDEP}] =dev-python/packaging-15.3[${PYTHON_USEDEP}] =dev-python/retrying-1.3.3[${PYTHON_USEDEP}] virtual/python-ipaddress[${PYTHON_USEDEP}] " RDEPEND="${VENDOR_DEPEND} =dev-python/setuptools-18.2[${PYTHON_USEDEP}] "

sigmavirus24 commented 8 years ago

@xavfernandez ah I had missed that. I was more focused on whether urllib3 had been debundled from requests, not whether pip's dependencies had been debundled from pip.

@JohannesBuchner can you tell us what version of requests was installed? And they definitely have been debundled. I can tell by the traceback.


I'm looking at pip and https://github.com/pypa/pip/blob/c20ed50a82d7f8af793c9754f3c936794aa0b56d/pip/download.py#L323 seems to be the only place it creates a Retry object. I have to wonder if the isinstance check in urllib3 will not work when requests is debundled.

JohannesBuchner commented 8 years ago

I have requests 2.8.1

sigmavirus24 commented 8 years ago

So yeah, I'm willing to bet that because of the vendored function in pip._vendor we have a problem where isinstance checks fail. I wonder if this would be fixed by appending to that list

    vendored('requests.packages')
    vendored('requests.packages.urllib3')
    # urllib3's subpackages

@JohannesBuchner to confirm my suspicion can you run the following code?

from pip._vendor.requests.packages import urllib3
from requests.packages import urllib3 as _urllib3

r = urllib3.Retry(total=10)
if isinstance(r, _urllib3.Retry):
    print('@sigmavirus24 is wrong')
else:
    print('@sigmavirus24 is right')

If that ends up being correct, I'm going to write a quick patch and see if applying it helps you @JohannesBuchner because this will be a problem for other distros too (to some extent) probably.

sigmavirus24 commented 8 years ago

(FWIW, my guess about needing to specify requests.packages.*.*.* is only because I see six, six.moves, and six.moves.urllib there so I'm extrapolating just a tiny bit.)

JohannesBuchner commented 8 years ago

The code output says you are right.

JohannesBuchner commented 8 years ago

I doubt I will be able to reproduce the conditions of the network failure.

sigmavirus24 commented 8 years ago

Ah, that's true. @Lukasa thoughts on how to potentially reproduce the network failure?

Lukasa commented 8 years ago

Easiest way is to patch the socket module to force it to throw that particular error from getaddrinfo.

sigmavirus24 commented 8 years ago

Actually, @JohannesBuchner if you can apply the patch in https://github.com/pypa/pip/pull/3287 to your installation and then run

from pip._vendor.requests.packages import urllib3
from requests.packages import urllib3 as _urllib3

r = urllib3.Retry(total=10)
if isinstance(r, _urllib3.Retry):
    print('The patch fixed this')
else:
    print('The patch didn't fix this')

That should be confirmation enough for me since I'm not sure we can repro this with pip's current testsuite.