ImportError: cannot import name connection

[phdegnan]

This is related to https://github.com/sigmavirus24/requests-toolbelt/issues/82 - which appeared to have been fixed by 0.5.1 - but perhaps some of my installed versions are old. Installation and system details are below.

I'm trying to get the MG-RAST API running (https://github.com/MG-RAST/MG-RAST-Tools). I had to install requests-toolbelt which seemed to go fine:

$ easy_install --prefix=/data/software/PyM requests-toolbelt
Creating /data/software/PyM/lib/python2.7/site-packages/site.py
Searching for requests-toolbelt
Reading https://pypi.python.org/simple/requests-toolbelt/
Best match: requests-toolbelt 0.5.1
Downloading https://pypi.python.org/packages/source/r/requests-toolbelt/requests-toolbelt-0.5.1.tar.gz#md5=580ab16edf9d1afad883623ba7873af9
Processing requests-toolbelt-0.5.1.tar.gz
Writing /tmp/easy_install-Nf_fke/requests-toolbelt-0.5.1/setup.cfg
Running requests-toolbelt-0.5.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-Nf_fke/requests-toolbelt-0.5.1/egg-dist-tmp-bonI2S
no previously-included directories found matching 'docs/_build'
warning: no previously-included files matching '*.py[cdo]' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no previously-included files matching '*.so' found anywhere in distribution
warning: no previously-included files matching '*.pyd' found anywhere in distribution
zip_safe flag not set; analyzing archive contents...
Adding requests-toolbelt 0.5.1 to easy-install.pth file

Installed /data/software/PyM/lib/python2.7/site-packages/requests_toolbelt-0.5.1-py2.7.egg
Processing dependencies for requests-toolbelt
Finished processing dependencies for requests-toolbelt

Path variable updates were added for local directories and MG-RAST-Tools:

export PYTHONPATH="/data/software/PyM/lib/python2.7/site-packages/:/data/software/MG-RAST-Tools/tools/lib/"
export PERL5LIB="/data/software/PM/share/perl/5.18.2:/data/software/PM/lib/perl/5.18.2/:/data/software/MG-RAST-Tools/tools/lib/"

But when I run the main script I get the following error message:

$ ./tools/bin/mg-download.py -h
Traceback (most recent call last):
  File "./tools/bin/mg-download.py", line 7, in <module>
    from mglib import *
  File "/data/software/MG-RAST-Tools/tools/lib/mglib.py", line 14, in <module>
    from requests_toolbelt import MultipartEncoder
  File "build/bdist.linux-x86_64/egg/requests_toolbelt/__init__.py", line 12, in <module>

  File "build/bdist.linux-x86_64/egg/requests_toolbelt/adapters/__init__.py", line 12, in <module>

  File "build/bdist.linux-x86_64/egg/requests_toolbelt/adapters/ssl.py", line 14, in <module>

  File "build/bdist.linux-x86_64/egg/requests_toolbelt/_compat.py", line 20, in <module>
ImportError: cannot import name connection

Per #82 my system installation is thus:

$  lsb_release -a 
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.3 LTS
Release:    14.04

$ pip list
apt-xapian-index (0.45)
argparse (1.2.1)
biom-format (1.3.1)
chardet (2.0.1)
cogent (1.5.3)
colorama (0.2.5)
configobj (4.7.2)
cutadapt (1.2.1)
docutils (0.11)
emperor (0.9.3)
gdata (2.0.18)
html5lib (0.999)
Jinja2 (2.7.2)
Landscape-Client (14.12)
MarkupSafe (0.18)
matplotlib (1.3.1)
mpi4py (1.3.1)
nose (1.3.1)
numpy (1.7.1)
openpyxl (1.7.0)
PAM (0.4.2)
Pillow (2.3.0)
pip (1.5.4)
prettytable (0.7.2)
pycrypto (2.6.1)
pycups (1.9.66)
pycurl (7.19.3)
Pygments (1.6)
pygobject (3.12.0)
PyNAST (1.2)
pyOpenSSL (0.13)
pyparsing (2.0.1)
pyqi (0.3.1)
pyserial (2.6)
pysmbc (1.0.14.1)
python-apt (0.9.3.5ubuntu2)
python-dateutil (1.5)
python-debian (0.1.21-nmu2ubuntu2)
pytz (2012c)
qcli (0.1.0)
reportlab (3.0)
requests (2.2.1)
requests-toolbelt (0.5.1)
roman (2.0.0)
setuptools (3.3)
six (1.5.2)
Sphinx (1.2.2)
ssh-import-id (3.21)
system-service (0.1.6)
tax2tree (1.0)
Twisted-Core (13.2.0)
urllib3 (1.7.1)
wheel (0.24.0)
wsgiref (0.1.2)
zope.interface (4.0.5)

$ pip show requests

---
Name: requests
Version: 2.2.1
Location: /usr/lib/python2.7/dist-packages
Requires: 

$ pip show requests-toolbelt

---
Name: requests-toolbelt
Version: 0.5.1
Location: /data/software/PyM/lib/python2.7/site-packages/requests_toolbelt-0.5.1-py2.7.egg
Requires: requests

I haven't tried updating requests based on pbkhrv's in #82 that it broke pip. Any suggestions would be great. Best, Patrick

[sigmavirus24]

I haven't had a chance to look into this yet but I just want to sincerely thank you for such a wonderfully detailed bug report. :cake:

[sigmavirus24]

So I refreshed my memory of the code that might be triggering this exception.

Your version of requests is supported (we test against 2.2.1 explicitly), but I'm a bit surprised we haven't run into this before. urllib3 1.7.1 is the actual culprit here as it did not include the urllib3.connection module until a later release.

So we'll need to figure out how to get what we need from those versions of urllib3. As best I can tell, the only thing we need from the urllib3.connection module is the HTTPConnection class and that's only for a feature we only support for versions of requests >= 2.4.0.

It shouldn't be terribly difficult to only conditional perform that import in requests_toolbelt._compat. Since the module is private we have no backwards compatibility guarantees to users.

Would you like to try doing that @phdegnan?

[phdegnan]

If you can tell me what precisely to do I'm happy to try. When it comes to coding I'm not even semi-pro.

It also seems that I could update requests to achieve a similar result?

[sigmavirus24]

So you'll need to edit 2 files:

• requests_toolbelt/_compat.py
• requests_toolbelt/adapters/socket_options.py

In the first you'll want the code to look something like:

# requests_toolbelt/_compat.py
import requests

if requests.__build__ >= 0x020400:
    try:
        from requests.packages.urllib3 import connection
    except ImportError:
        from urllib3 import connection
else:
    connection = None

And you want to remove the lines from this try/except block that match the ones above.

Then you'll want to edit this block to look like

# requests_toolbelt/adapters/socket_options.py

class SocketOptionsAdapter(adapters.HTTPAdapter):
    """ ....

    """

    if connection is not None:
        # What is presently there
    else:
        default_socket_options = [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)]

[phdegnan]

Great - I'll give it a whirl and get back to you. Cheers!

[sigmavirus24]

Awesome! Thanks for reporting & fixing this!

[sigmavirus24]

Side note: The reason our testing didn't catch this on 2.2.1 is because requests had a version that was newer than 1.7.1 bundled which did include the urllib3.connection module. Even 2.1.0 has the connection.py file (and so does 2.1.0).

So the fault for this is squarely on the distro packaging. This is very disappointing. Still I'd rather handle this for users like @phdegnan than ignore it as a problem out of our control. (I just wish the distributions weren't this awful with the packages they provide users.)

[phdegnan]

Well, their code compiles without error.

$ ./tools/bin/mg-download.py -h
NAME
    mg-download

VERSION
    1

SYNOPSIS
    mg-download [ --help, --user <user>, --passwd <password>, --token <oAuth token>, --project <project id>, --metagenome <metagenome id>, --file <file id> --dir <directory name> --list <list files for given id>]
...

Actually retrieving data from the MG-RAST server seems ungodly slow. Hopefully that is their fault...

Thanks for the timely assistance!

[sigmavirus24]

@phdegnan are you going to send a PR?

[phdegnan]

I can.

[pensnarik]

I have the same issue in package, provided in Ubuntu Server 14.04. Fix, described by you doesn't helps. The error now is:

ImportError: No module named packages.urllib3.poolmanager

Would it be fixed?

[sigmavirus24]

@pensnarik how did you install requests and the requests toolbelt?