_NamespacePath object has no attribute sort (31.0.0)

[jkbbwr]

When pip installing a package that shares a namespace it fails out on 31.0.0 but installs fine on 28.8.0

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/usr/lib/python3.5/site-packages/setuptools/__init__.py", line 10, in <module>
        from setuptools.extern.six.moves import filter, filterfalse, map
      File "/usr/lib/python3.5/site-packages/setuptools/extern/__init__.py", line 1, in <module>
        from pkg_resources.extern import VendorImporter
      File "/usr/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3015, in <module>
        @_call_aside
      File "/usr/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2999, in _call_aside
        f(*args, **kwargs)
      File "/usr/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3043, in _initialize_master_working_set
        for dist in working_set
      File "/usr/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3043, in <genexpr>
        for dist in working_set
      File "/usr/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2573, in activate
        declare_namespace(pkg)
      File "/usr/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2147, in declare_namespace
        _handle_ns(packageName, path_item)
      File "/usr/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2087, in _handle_ns
        _rebuild_mod_path(path, packageName, module)
      File "/usr/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2116, in _rebuild_mod_path
        orig_path.sort(key=position_in_sys_path)
    AttributeError: '_NamespacePath' object has no attribute 'sort'

[tseaver]

From what I can tell, this is a 3.5-only failure.

[hciudad]

I can trigger the error just by importing setuptools. However, when I do it a second time, it imports fine. So, if I do something like this, the install succeeds.

try:
    from setuptools import setup, find_packages
except AttributeError:
    from setuptools import setup, find_packages

[jaraco]

I tried adding a test to capture the failure, but the test is passing. So to some extent it's working. Why is the test working but your package not?

[tseaver]

@jaraco The bug happens trying to import setuptools/pkg_resources in an environment where a namespace package is installed, but where there is more than one directory in the path of the namespace package (e.g., one package in a "development install" vs. dependencies installed into site-packages).

In @jkbbwr's case above, what would cause zope.__path__ to be an instance of _NamespacePath rather than the plain list that the code in pkg_resources._rebuild_mod_path clearly expects? That class appears only to be instantiated by the stdlib's import machinery under Python 3.5+:

• importlib._bootstrap_external._NamespaceLoader.__init__
• importlib._bootstrap_external.PathFinder.find_spec.

As a hackaround, one might add monkeypatch a sort method onto importlib._bootstrap_external._NamespacePath. ;)

[jaraco]

I've just pushed another commit, this one in the test_develop module. This test does as you describe, installs one package into a simulated site-packages (myns.pkgA) and another using develop (myns.pkgB). Both packages are importable and pkg_resources can still be imported as well.

[jaraco]

what would cause zope.__path__ to be an instance of _NamespacePath rather than the plain list that the code in pkg_resources._rebuild_mod_path clearly expects?

It's almost certainly the fact that the new -nspkg.pth files are using importlib.util.module_from_spec to initialize the module (and thus setting __path__).

I continue to be unable to replicate the error indicated. Here's one such attempt:

$ python -m venv ~/.envs/issue885                      
$ ~/.envs/issue885/bin/pip install -U setuptools       
Collecting setuptools
  Using cached setuptools-31.0.0-py2.py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
Successfully installed setuptools-31.0.0
$ ~/.envs/issue885/bin/pip install -U zope.component
Collecting zope.component
  Using cached zope.component-4.3.0.tar.gz
Requirement already up-to-date: setuptools in /Users/jaraco/.envs/issue885/lib/python3.6/site-packages (from zope.component)
Collecting zope.interface>=4.1.0 (from zope.component)
  Using cached zope.interface-4.3.2.tar.gz
Collecting zope.event (from zope.component)
  Using cached zope.event-4.2.0.tar.gz
Installing collected packages: zope.interface, zope.event, zope.component
  Running setup.py install for zope.interface ... done
  Running setup.py install for zope.event ... done
  Running setup.py install for zope.component ... done
Successfully installed zope.component-4.3.0 zope.event-4.2.0 zope.interface-4.3.2
$ pwd
/Users/jaraco/zope.interface
$ ~/.envs/issue885/bin/pip install -e .             
Obtaining file:///Users/jaraco/zope.interface
Requirement already satisfied: setuptools in /Users/jaraco/.envs/issue885/lib/python3.6/site-packages (from zope.interface==4.3.3.dev0)
Installing collected packages: zope.interface
  Found existing installation: zope.interface 4.3.2
    Uninstalling zope.interface-4.3.2:
      Successfully uninstalled zope.interface-4.3.2
  Running setup.py develop for zope.interface
Successfully installed zope.interface
$ cd ~
$ ~/.envs/issue885/bin/python
Python 3.6.0rc1 (v3.6.0rc1:29a273eee9a5, Dec  6 2016, 16:24:13) 
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkg_resources
>>> import zope.interface
>>> zope.interface.__file__
'/Users/jaraco/zope.interface/src/zope/interface/__init__.py'
>>> zope.__path__
_NamespacePath(['/Users/jaraco/.envs/issue885/lib/python3.6/site-packages/zope', '/Users/jaraco/zope.interface/src/zope'])

What steps do you have to invoke to get a virtual environment that replicates this issue?

[theacodes]

This is also occurring consistently over at https://github.com/GoogleCloudPlatform/google-auth-library-python in our lint tox (sample failure).

It'll also occur in other tox environment if you run them twice. The first time tox creates the environment from scratch, but when it tries to re-use an existing environment it 'splodes.

Let me know if I can do anything else to help reproduce/fix this.

[jaraco]

Excellent. That does help. I simply checked out GoogleCloudPlatform/google-auth-library-python and ran tox -e lint and that replicated the failure. I should be able to distill the issue from there. Thanks.

[theacodes]

Awesome. 👍

[warsaw]

I just hit this same problem. I can craft a reproducer if you need it. I see it when running tox on a py35 environment with flufl.i18n, which uses flufl.testing, both of which are namespace packages. I need to commit and push a fix to the latter for you to reproduce it. Let me know if you need it, but it would be really great to get a quick fix released for this bug.

[warsaw]

FWIW, it affects Python 3.6 also; I can't test 3.4.

[jaraco]

Working with the google-auth library, I find that if I set usedevelop = True in the tox settings, the issue doesn't occur. This behavior indicates to me that the issue lies in part with a package which is duplicately installed. In any case, I was able to use it to create a test case that captures the error.

[warsaw]

@jaraco That's interesting because I have usedevelop=True and it happens for me. But at least you ahve a reproducer. :)

[theacodes]

usedevelop probably won't work except in that specific case because google-auth uses namespace packages.

On Tue, Dec 13, 2016, 6:01 PM Jason R. Coombs notifications@github.com wrote:

Working with the google-auth library, I find that if I set usedevelop = True in the tox settings, the issue doesn't occur. This behavior indicates to me that the issue lies in part with a package which is duplicately installed. In any case, I was able to use it to create a test case that captures the error.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pypa/setuptools/issues/885#issuecomment-266921037, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPUcwf9-z3hwrnuPlbbGLeyiea38dH-ks5rH04MgaJpZM4LKdo7 .

[jaraco]

@jonparrott: The whole point of v31.0.0 and the fix for #250 is that 'develop' and namespace packages should work well together now, so I hope you can drop that assumption soon.

@warsaw: The fix is in place. Can you update setuptools in your test environment to git+https://github.com/pypa/setuptools@3943cf4 and see if that addresses the manifestation you encountered?

[jaraco]

Well hell. The fix works for Python 3.6 but not for Python 3.5. Thanks to Travis for catching that.

[jaraco]

Okay. Yet another version to test against, this one passes on Python 3.5. git+https://github.com/pypa/setuptools@7c0c39ef. I'm not happy with the fix, but it passes the tests.

[warsaw]

@jaraco I get another bug now:

Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    from setuptools import setup, find_packages
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/setuptools/__init__.py", line 10, in <module>
    from setuptools.extern.six.moves import filter, filterfalse, map
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/setuptools/extern/__init__.py", line 1, in <module>
    from pkg_resources.extern import VendorImporter
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3015, in <module>
    @_call_aside
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2999, in _call_aside
    f(*args, **kwargs)
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3043, in _initialize_master_working_set
    for dist in working_set
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 3043, in <genexpr>
    for dist in working_set
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2573, in activate
    declare_namespace(pkg)
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2147, in declare_namespace
    _handle_ns(packageName, path_item)
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2087, in _handle_ns
    _rebuild_mod_path(path, packageName, module)
  File "/home/barry/projects/flufl/i18n/.tox/py35-nocov/lib/python3.5/site-packages/pkg_resources/__init__.py", line 2116, in _rebuild_mod_path
    orig_path[:] = sorted(orig_path, key=position_in_sys_path)
TypeError: '_NamespacePath' object does not support item assignment

[jaraco]

@warsaw: For some reason, I didn't get notified of your message. My latest commit should have addressed the item assignment bug (by simply bypassing the sort behavior). I've released the fix as v31.0.1. Please let me know how that works for you.

[tseaver]

@jaraco I can confirm that 31.0.1 fixes the Python 3.5 test failure for google-cloud-python.

[warsaw]

@jaraco Something seems very messed up with my tox environments, but if I wrestle it enough to install the latest pypi version, it does seem to work well now. Thanks!

[theacodes]

Thanks, @jaraco!

[villasv]

I'm having this issue when trying to pip install --upgrade git+ssh://git@github.com/some/package.git#egg=name when the package is already installed (trying to upgrade).

pip list shows setuptools (32.1.2)

Should I file a new issue?

[jaraco]

@villasv: You probably have an old version of pkg_resources lingering in your environment. First check that import pkg_resources; print(pkg_resources.__file__) doesn't point to a lingering old version. If that's not the issue, use pip to uninstall and reinstall setuptools. If you still have the issue, do please file a new issue with steps to replicate.

[CPlusPlus17]

Broken with: 34.1.1

[GordianDziwis]

I have got the same error with pip form the distribution python3-pip packages from ubuntu 16.10 If I execute import pkg_resources; print(pkg_resources.__file__) in the shell the first time it throws the error from op, the second time it works.

[jaraco]

This issue is fixed in setuptools, but it's still broken in pip.

[jaraco]

As this issue is still affecting me in pip, I've continued to investigate. I was able to replicate the error simply by installing a namespace package, having that namespace package also minimally present on sys.path (in .) and to import pkg_resources, e.g.:

pip install jaraco.itertools
mkdir jaraco
echo '__import__("pkg_resources").declare_namespace(__name__)' > jaraco/__init__.py
python -c "import pkg_resources"

Note that the jaraco directory need not have the declare_namespace When initializing the master working set, pkg_resources activates the jaraco.itertools distribution, which causes the declare_namespace('jaraco'), which fails on the _handle_ns('jaraco', '').

The issue only occurs when the installed namespace package was built with Setuptools 31.0.0 or later (with the newest -nspkg.pth file technique). I don't encounter the issue if installing an older wheel that doesn't have that newer technique.

[jaraco]

Tracing the code, the reason the aberrant behavior only exhibits when the -nspkg.pth was built on Setuptools 31 is because the old -nspkg.pth behavior would only add modules to sys.modules for older Pythons and would rely on pep-420 otherwise.

As a result, in Distribution.activate, the call to declare_namespace would not be reached.

With a Setuptools 31 -nspkg.pth file, declare_namespace gets called, which invokes _handle_ns on every path in sys.path, including ''.

[esafak]

@tseaver setuptools 31.0.1 did not work for me. What version of pip are you on? Does anyone know a combination of setuptools, pip, and google-cloud (recent version) that works reliably?

[tseaver]

@esafak Try updating both setuptools and pip in a new virtualenv:

$ python3 -m venv /tmp/gcloud
$ /tmp/pypa-885/bin/pip3 install -U setuptools pip
...
$ /tmp/pypa-885/bin/pip3 install google-cloud
...

And then:

>>> from google.cloud.datastore import Client

[thijstriemstra]

Also ran into the issue on readthedocs with python 3.5: https://readthedocs.org/projects/luma-led-matrix/builds/5081789/ but it's possible to use python 2.7 there which presumably doesn't have this issue, so I think I'll take the easy way out.

[palewire]

This bug has popped up again over here in this ticket https://github.com/opencivicdata/python-opencivicdata-django/issues/80

The user reports:

Python 3.5.2 setuptools 34.3.2

[aronhelser]

I can repro this issue very simply, python 3.6, setuptools 34.3.3, virtualenv 15.1.0, windows10, mingw64 git bash shell

virtualenv runtime source runtime/Scripts/active pip install twisted pip install autobahn

this installs twisted 17.1.0, and then fails trying to install autobahn

[ye]

I was having the similar errors while trying to install google cloud sdk while building a docker image with python 3.5.2, pip 9.0.1, setuptools 34.3.2 (which is based off official python:3.5-slim docker image). Logs is here.

Force pip upgrade to setuptools 34.3.3 made it work, one liner workaround:

RUN pip install --upgrade setuptools==34.3.3 && pip install google-cloud

And it appears that pypi.python.org has already been updated to offer setuptools 34.3.3 as the latest version.

[jaraco]

I can repro this issue very simply,

Oy. That's on Windows using an open-source compiler and a couple of different other packages. I did try the same on MacOS with Python 3.6.1 but couldn't replicate the issue, so something about Windows or mingw64 is apparently implicated. Plus, we know that pip still bundles this bug, so that's probably the issue.

[aronhelser]

Thanks for trying my setup out!

I am not seeing the issue with my main install of python 3.6 - only inside the virtualenv. Does that make sense with your observation that 'pip still bundles this bug?'

It's even more fragile than I thought - after the first 'pip install twisted', I can't even do 'pip list' - I get the same error. File "c:\akit\runtime\lib\site-packages\pip\_vendor\pkg_resources\__init__.py", line 2121, in _rebuild_mod_path orig_path.sort(key=position_in_sys_path) AttributeError: '_NamespacePath' object has no attribute 'sort'

Setuptools is listed as version 34.3.3 inside the virtualenv lib/site-packages.

I hope this info is helpful - let me know if there's anything I can test which might confirm/deny?

[aronhelser]

BTW, one of Twisted's dependencies is zope.interface, which I saw listed as a problem package in one of these referenced bugs, I think?

[FRidh]

yep, zope.interface is also a namespace package where we experience problems with.

[jaraco]

Just look at the trace back you get. If the error is occurring in a file found in pip, then the issue doesn't lie with Setuptools, and it won't matter which version of Setuptools you have installed.

[aronhelser]

Yes, pip list has a traceback totally within pip. Thank you for clarifying it for me. I see the pip bug https://github.com/pypa/pip/issues/4216 is still open and they haven't quite managed to get a fix out yet.

[dhinakaransdk]

It worked for me just run

conda config --set ssl_verify False

if you are facing this issue while creating an environment on anaconda 3.6.1

[SeungKil-Paek]

I modified the function _rebuild_mod_path of the file(.../site-packages/pip/_vendor/pkg_resources/__init__.py), which worked:

orig_path_t = list(orig_path) # added
#orig_path.sort(key=position_in_sys_path) # commented
orig_path_t.sort(key=position_in_sys_path) # added
#module.__path__[:] = [_normalize_cached(p) for p in orig_path] # commented
module.__path__[:] = [_normalize_cached(p) for p in orig_path_t] # added

[tseaver]

An even better fix would be to use sorted, e.g.:

module.__path__[:] = [_normalize_cached(p) for p in sorted(orig_path)]

[jonathan-kosgei]

Fixed this by upgrading my setuptools

pip install --upgrade setuptools

[lopezco]

I've still this bug on the latest version (36.0.1) whereas it works fine on the previous release on pypi (35.0.2)

[ispmarin]

Same problem here. Ubuntu 16.04, python 3.5 inside a virtualenv, setuptools 36.0.1 gives this error while 35.0.2 works fine.

[jaraco]

You probably aren't getting this with Setuptools, but with pkg_resources vendored in pip. If you are getting the error in setuptools, please file a new ticket and include the traceback and other environmental conditions that created the bug. Otherwise, look for the pip link above and follow that.

[tjmills]

Also fixed this issue with an update of setup tools by first pip uninstalling setup tools and reinstalling

[frankyaorenjie]

fixed by pip install --upgrade setuptools pip install --upgrade pip