Latest Python Module Packages No Longer Visible to pip

[jsetton]

Package: python-idna

Prior to the latest update (September '17), the Python modules installed via opkg were visible to pip. When installing the latest version, pip no longer sees the installed packages and tries to install from the Python package repository instead, which on devices with no available compiling tools is a big problem. When looking at the files differences for a given module, I noticed that the dist or egg-info files have been removed in the latest version preventing, most likely, pip from tracking these installed modules.

I use python-idna as an example.

$ wget http://pkg.entware.net/binaries/mipsel/archive/python-idna_2.1-1_mipselsf.ipk
[...]
$  opkg install ./python-idna_2.1-1_mipselsf.ipk
Installing python-idna (2.1-1) to root...
$ opkg files python-idna | sort
/opt/lib/python2.7/site-packages/idna-2.1-py2.7.egg-info/PKG-INFO
/opt/lib/python2.7/site-packages/idna-2.1-py2.7.egg-info/SOURCES.txt
/opt/lib/python2.7/site-packages/idna-2.1-py2.7.egg-info/dependency_links.txt
/opt/lib/python2.7/site-packages/idna-2.1-py2.7.egg-info/top_level.txt
/opt/lib/python2.7/site-packages/idna/__init__.pyc
/opt/lib/python2.7/site-packages/idna/codec.pyc
/opt/lib/python2.7/site-packages/idna/compat.pyc
/opt/lib/python2.7/site-packages/idna/core.pyc
/opt/lib/python2.7/site-packages/idna/idnadata.pyc
/opt/lib/python2.7/site-packages/idna/intranges.pyc
/opt/lib/python2.7/site-packages/idna/uts46data.pyc
Package python-idna (2.1-1) is installed on root and has the following files:
$ pip list | grep idna
idna (2.1)

$ opkg install python-idna
Upgrading python-idna on root from 2.1-1 to 2.5-1...
Downloading http://pkg.entware.net/binaries/mipsel/python-idna_2.5-1_mipselsf.ipk.
Removing obsolete file /opt/lib/python2.7/site-packages/idna-2.1-py2.7.egg-info/PKG-INFO.
Removing obsolete file /opt/lib/python2.7/site-packages/idna-2.1-py2.7.egg-info/dependency_links.txt.
Removing obsolete file /opt/lib/python2.7/site-packages/idna-2.1-py2.7.egg-info/top_level.txt.
Removing obsolete file /opt/lib/python2.7/site-packages/idna-2.1-py2.7.egg-info/SOURCES.txt.
Configuring python-idna.
$ opkg files python-idna | sort
/opt/lib/python2.7/site-packages/idna/__init__.pyc
/opt/lib/python2.7/site-packages/idna/codec.pyc
/opt/lib/python2.7/site-packages/idna/compat.pyc
/opt/lib/python2.7/site-packages/idna/core.pyc
/opt/lib/python2.7/site-packages/idna/idnadata.pyc
/opt/lib/python2.7/site-packages/idna/intranges.pyc
/opt/lib/python2.7/site-packages/idna/uts46data.pyc
Package python-idna (2.5-1) is installed on root and has the following files:
$ pip list | grep idna

$ opkg list-installed python-pip
python-pip - 9.0.1
$ which pip
/opt/bin/pip
$ pip --version
pip 9.0.1 from /opt/lib/python2.7/site-packages (python 2.7)

Platform:

Asus RT-AC66U Linux 2.6.22.19 mips GNU/Linux

[zyxmon]

An option to keep egg files was added upstream after the last enware update - https://github.com/openwrt/packages/commit/074d2863be48c1a9093fffcda91f7a328bc7d0cb

We will add this on next sync.

[jsetton]

Great. Thanks for the update.

[zyxmon]

The upcoming version has the same behaviour. Probably you should report this upstream.

[jsetton]

@zyxmon thanks for the update although I am confused of this outcome since you provided the change added upstream above, which also was merged. How were you able to come to this conclusion? Is there another issue opened that you are looking at?

[zyxmon]

I have just checked the upcoming python update. It has the change added upstream. It has a bug. IMHO package installation does not have some files/info need for pip to list installed modules,

[jsetton]

Do you think you could list the files part of the package I used as an example above for the upcoming version? I will report this upstream as you recommended using that information. Thanks.

[jsetton]

So I took a look at this again and I am wondering if the new upstream keep egg files option was enabled during the last entware-ng build? As far as I can see, this option (PYTHON_KEEP_EGGINFO) defaults to off if not specified which would explain why the behavior didn't change.

[zyxmon]

We will check. Probably we will change PYTHON_KEEP_EGGINFO in https://github.com/Entware-for-kernel-3x/entware-packages-3x/blob/master/lang/python/python/files/python-package.mk#L35 next build in Entware-3x end Entware-ng.

[jsetton]

Thanks