namespaced dependencies cannot be found in a custom PYTHONPATH

[kodawah]

Describe the bug Using the latest stable version of pip, I am trying to install twisted (which depends on zope.interface), on a custom PYTHONPATH location.

pip install --ignore-installed --prefix=$STACK_PREFIX twisted

This env variable is set to

$ echo $PYTHONPATH /home/user/stack/lib/python3.6/site-packages:/home/user/stack/lib64/python2.7/site-packages/:/home/user/stack/lib/python2.7/site-packages

and both twisted and and zope.interface are installed correctly

$ find stack -name zope stack/lib64/python2.7/site-packages/zope $ ls stack/lib64/python2.7/site-packages/zope interface $ find stack -name twisted stack/lib64/python2.7/site-packages/twisted

The package twisted can be loaded just fine -- zope.interface however cannot be found

Python 2.7.5 (default, Apr  2 2020, 13:16:51) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twisted
>>> import zope.interface
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ImportError: No module named zope.interface

Following the advice from here https://github.com/pyinstaller/pyinstaller/issues/502 if I manually add an empty __init__.py to stack/lib64/python2.7/site-packages/zope then the module becomes discoverable

$ touch stack/lib64/python2.7/site-packages/zope/__init__.py 
$ ls  stack/lib64/python2.7/site-packages/zope
__init__.py  interface
$ python
Python 2.7.5 (default, Apr  2 2020, 13:16:51) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import twisted
>>> import zope.interface
>>> 

Expected behavior I don't think it should be required to manually add the init file to the PYTHONPATH location. Even this is a namespaced module it should be discoverable in the custom PYTHONPATH location like any other module.

To Reproduce

1. set PYTHONPATH to a custom location
2. install twisted pip install --ignore-installed --prefix=$STACK_PREFIX twisted
3. import zope.interface in python

My Platform

$pip -V pip 20.1.1 from /home/user/stack/lib/python2.7/site-packages/pip (python 2.7) $ uname -a Linux b582ede63e34 4.19.76-linuxkit pypa/warehouse#1 SMP Tue May 26 11:42:35 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[uranusjr]

Without looking deep into it, this feels like a Python import logic issue, since pip does not handle namespace packages specially in any way afaik.

[kodawah]

@uranusjr so the namespace packages just happen to work when they are not in a custom PYTHONPATH?

I'm not too familiar with python import logic, should I post a ticket somewhere else? Thanks for your advice

[uranusjr]

One important thing to do here is to reduce the example to minimise the moving parts in it, and determine whether pip or Python is at fault here. The reduced example should contain as little as possible, and be reliably reproducible from scratch.

For example, this is what I tried from reading your description:

$ python3 -m pip install twisted --target ./target
Collecting twisted
  Downloading Twisted-20.3.0-cp37-cp37m-macosx_10_6_intel.whl (3.1 MB)
     |████████████████████████████████| 3.1 MB 605 kB/s 
Collecting constantly>=15.1
  Downloading constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting PyHamcrest!=1.10.0,>=1.9.0
  Downloading PyHamcrest-2.0.2-py3-none-any.whl (52 kB)
     |████████████████████████████████| 52 kB 135 kB/s 
Collecting incremental>=16.10.1
  Downloading incremental-17.5.0-py2.py3-none-any.whl (16 kB)
Collecting Automat>=0.3.0
  Downloading Automat-20.2.0-py2.py3-none-any.whl (31 kB)
Collecting hyperlink>=17.1.1
  Downloading hyperlink-19.0.0-py2.py3-none-any.whl (38 kB)
Collecting zope.interface
  Downloading zope.interface-5.1.0-cp37-cp37m-macosx_10_9_x86_64.whl (192 kB)
     |████████████████████████████████| 192 kB 383 kB/s
Collecting attrs>=19.2.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting six
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting idna>=2.5
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
     |████████████████████████████████| 58 kB 649 kB/s 
Collecting setuptools
  Downloading setuptools-47.3.1-py3-none-any.whl (582 kB)
     |████████████████████████████████| 582 kB 375 kB/s
Installing collected packages: constantly, PyHamcrest, incremental, attrs, six, Automat, idna, hyperlink, setuptools, zope.interface, twisted
Successfully installed Automat-20.2.0 PyHamcrest-2.0.2 attrs-19.3.0 constantly-15.1.0 hyperlink-19.0.0 idna-2.10 incremental-17.5.0 setuptools-47.3.1 six-1.15.0 twisted-20.3.0 zope.interface-5.1.0
$ PYTHONPATH=$PWD/target python3 -c 'import zope.interface; print(zope.interface)'
<module 'zope.interface' from '/.../target/zope/interface/__init__.py'>

Everything installs and imports correctly, so it seems that neither --target nor the namespace package is at fault here. This means that there is something in the setup not mentioned in your description that makes things fail. You need to hunt that thing down first by adding/tweaking things in the setup piece by piece.

[kodawah]

hi, thanks for your suggestion, I tried to minimize the moving parts in the "Steps to reproduce" section

set PYTHONPATH to a custom location
install twisted pip install --ignore-installed --prefix=$STACK_PREFIX twisted
import zope.interface in python

You may note that i used --ignore-installed and --prefix instead of --target - I tried your script and i can confirm it works. Here are the two command lines to hopefully reproduce the reported problem on your env as well

$ python3 -m pip install twisted --ignore-installed --prefix=./target
$ PYTHONPATH=$PWD/target python3 -c 'import zope.interface; print(zope.interface)'

which result in

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'zope'

[jankatins]

Not sure if this a new problem or just the same:

[15:59:24] λ  rm -rf /tmp/zope_problem

[15:59:44] λ  /home/jan.katins/.virtualenvs/pyinfra/bin/python3 -m pip --disable-pip-version-check install --target /tmp/zope_problem zope.interface zope.event
Collecting zope.interface
  Using cached zope.interface-5.4.0-cp39-cp39-manylinux2010_x86_64.whl (255 kB)
Collecting zope.event
  Using cached zope.event-4.5.0-py2.py3-none-any.whl (6.8 kB)
Collecting setuptools
  Using cached setuptools-57.4.0-py3-none-any.whl (819 kB)
Installing collected packages: setuptools, zope.interface, zope.event
  WARNING: Value for scheme.platlib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
  distutils: /tmp/pip-target-irw4lnin/lib64/python
  sysconfig: /tmp/pip-target-irw4lnin/lib/python
  WARNING: Additional context:
  user = False
  home = '/tmp/pip-target-irw4lnin'
  root = None
  prefix = None
Successfully installed setuptools-57.4.0 zope.event-4.5.0 zope.interface-5.4.0
WARNING: Target directory /tmp/zope_problem/zope already exists. Specify --upgrade to force replacement.

[15:59:55] λ  find  /tmp/zope_problem/ |grep /zope/ |grep "\.py$"
/tmp/zope_problem/zope/event/tests.py
/tmp/zope_problem/zope/event/classhandler.py
/tmp/zope_problem/zope/event/__init__.py

[16:00:04] λ  /home/jan.katins/.virtualenvs/pyinfra/bin/python3 -m pip --disable-pip-version-check install --target /tmp/zope_problem zope.interface zope.event --upgrade
Collecting zope.interface
  Using cached zope.interface-5.4.0-cp39-cp39-manylinux2010_x86_64.whl (255 kB)
Collecting zope.event
  Using cached zope.event-4.5.0-py2.py3-none-any.whl (6.8 kB)
Collecting setuptools
  Using cached setuptools-57.4.0-py3-none-any.whl (819 kB)
Installing collected packages: setuptools, zope.interface, zope.event
  WARNING: Value for scheme.platlib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
  distutils: /tmp/pip-target-3u8y7xfp/lib64/python
  sysconfig: /tmp/pip-target-3u8y7xfp/lib/python
  WARNING: Additional context:
  user = False
  home = '/tmp/pip-target-3u8y7xfp'
  root = None
  prefix = None
Successfully installed setuptools-57.4.0 zope.event-4.5.0 zope.interface-5.4.0

[16:00:17] λ  find  /tmp/zope_problem/ |grep /zope/ |grep "\.py$"
/tmp/zope_problem/zope/interface/tests/test_sorting.py
[...]
/tmp/zope_problem/zope/interface/_flatten.py

For whatever reason, only one namespace package content survives but the dist-info files are there for both:

[16:03:58] λ  ls  /tmp/zope_problem/ | grep zope
zope
zope.event-4.5.0.dist-info
zope.event-4.5.0-py3.6-nspkg.pth
zope.interface-5.4.0.dist-info
zope.interface-5.4.0-py3.9-nspkg.pth

Installing it without --target ... works as expected:

[16:06:18] λ  /home/jan.katins/.virtualenvs/pyinfra/bin/python3 -m pip --disable-pip-version-check install  zope.event zope.interface --upgrade
Requirement already satisfied: zope.event in ./.virtualenvs/pyinfra/lib/python3.9/site-packages (4.5.0)
Requirement already satisfied: zope.interface in ./.virtualenvs/pyinfra/lib/python3.9/site-packages (5.4.0)
Requirement already satisfied: setuptools in ./.virtualenvs/pyinfra/lib/python3.9/site-packages (from zope.event) (53.0.0)

~  🐍 v3.9.6 (venv:pyinfra)
[16:06:41] λ  ls  ./.virtualenvs/pyinfra/lib/python3.9/site-packages/zope
event  interface

[uranusjr]

The original report was specific to Python 2, so I'm going to close this. (It's a limitation in Python's import system and not related to pip.) @jankatins I am not sure if your issue is the same and you should probably open a new issue.