search

ros-infrastructure / rosdep

rosdep multi-package manager system dependency tool
http://ros.org/wiki/rosdep
BSD 3-Clause "New" or "Revised" License
77 stars 170 forks source link

pip doesn't find python packages installed in /opt/ros/indigo #456

Closed asmodehn closed 7 years ago

asmodehn commented 8 years ago

I haven't found the root cause of this problem yet, however this is my current case, which I find quite confusing (coming from a python background) : 

alexv@asmodehn:~$ python -c "import tornado; print tornado.__file__; print tornado.version"
/usr/lib/python2.7/dist-packages/tornado/__init__.pyc
3.1.1
alexv@asmodehn:~$ pip freeze | grep tornado
tornado==3.1.1
alexv@asmodehn:~$ source /opt/ros/indigo/setup.bash 
alexv@asmodehn:~$ python -c "import tornado; print tornado.__file__; print tornado.version"
/opt/ros/indigo/lib/python2.7/dist-packages/tornado/__init__.pyc
4.0.2
alexv@asmodehn:~$ pip freeze | grep tornado
tornado==3.1.1
asmodehn commented 8 years ago

rosdep installs python ros packages in /opt/ros/indigo (through ros debian packages), so maybe this is more of a "package structure" issue rather than rosdep itself ?

asmodehn commented 8 years ago

After more research, pip actually finds packages in /opt/ros/indigo : 

alexv@asmodehn:~$ pip list | grep ros
pyros-setup (0.1.0)
ros-buildfarm (1.1.0)
rosdep (0.11.4)
rosdistro (0.4.7)
rospkg (1.0.39)
alexv@asmodehn:~$ source /opt/ros/indigo/setup.bash 
alexv@asmodehn:~$ pip list | grep ros
gazebo-ros (2.4.10)
pyros-setup (0.0.8)
pyros-test (0.0.3)
ros-buildfarm (1.1.0)
rosapi (0.7.14)
rosbag (1.11.19)
rosboost-cfg (1.11.12)
rosbridge-library (0.7.14)
rosbridge-server (0.7.14)
rosclean (1.11.12)
roscreate (1.11.12)
rosdep (0.11.4)
rosdistro (0.4.7)
rosdoc-lite (0.2.6)
rosgraph (1.11.19)
roslaunch (1.11.19)
roslib (1.11.12)
roslint (0.10.0)
roslz4 (1.11.19)
rosmake (1.11.12)
rosmaster (1.11.19)
rosmsg (1.11.19)
rosnode (1.11.19)
rosparam (1.11.19)
rospkg (1.0.39)
rospy (1.11.19)
rospy-message-converter (0.4.0)
rosservice (1.11.19)
rostest (1.11.19)
rostopic (1.11.19)
rosunit (1.11.12)
roswtf (1.11.19)
smach-ros (2.0.0)
tf2-ros (0.5.13)
alexv@asmodehn:~$ pip freeze | grep tornado
tornado==3.1.1
alexv@asmodehn:~$ pip list | grep tornado
tornado (3.1.1)

So maybe the problem comes from somewhere else ?

tfoote commented 8 years ago

I'd suggest digging into how pip interacts with the PYTHONPATH. It may privilege things on standard paths over things inserted on the PYTHONPATH

asmodehn commented 7 years ago

@tfoote From my understanding PYTHONPATH is always a "last step" add on at the beginning of the sys.path list of paths where python modules/packages can be found. BTW this is different from the way ROS treat PYTHONPATH as the stack of workspaces (which kind of was decided against in python environment - venv alone, or system and venv, but no venv stacking mechanism). Also I wouldn't expect pip to behave in a different way than python does.

I digged a bit more... It seems this doesn't happen with a recent version of pip and a recent version of tornado package : 

alexv@AlexV-Linux:~$ pip --version
pip 8.1.2 from /usr/local/lib/python2.7/dist-packages (python 2.7)
alexv@AlexV-Linux:~$ pip list | grep ros
pyros-config (0.2.0)
pyros-setup (0.2.0)
rosdep (0.11.5)
rosdistro (0.6.1)
rosdistro-modules (0.6.1)
rosinstall (0.7.8)
rosinstall-generator (0.1.13)
rospkg (1.1.0)
rospkg-modules (1.1.0)
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

alexv@AlexV-Linux:~$ source /opt/ros/indigo/setup.bash 
alexv@AlexV-Linux:~$ pip list | grep ros
gazebo-ros (2.4.13)
pyros-config (0.2.0)
pyros-setup (0.2.0)
pyros-test (0.0.6)
pyros-utils (0.1.3)
rosbag (1.11.21)
rosboost-cfg (1.11.14)
rosclean (1.11.14)
roscreate (1.11.14)
rosdep (0.11.5)
rosdistro (0.6.1)
rosdistro-modules (0.6.1)
rosgraph (1.11.21)
rosinstall (0.7.8)
rosinstall-generator (0.1.13)
roslaunch (1.11.21)
roslib (1.11.14)
roslint (0.10.0)
roslz4 (1.11.21)
rosmake (1.11.14)
rosmaster (1.11.21)
rosmsg (1.11.21)
rosnode (1.11.21)
rosparam (1.11.21)
rospkg (1.1.0)
rospkg-modules (1.1.0)
rospy (1.11.21)
rospy-message-converter (0.4.0)
rosservice (1.11.21)
rostest (1.11.21)
rostopic (1.11.21)
rosunit (1.11.14)
roswtf (1.11.21)
tf2-ros (0.5.15)
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
alexv@AlexV-Linux:~$ pip list | grep tornado
tornado (4.2.1)
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
alexv@AlexV-Linux:~$ python -c "import tornado; print tornado.__file__"
/opt/ros/indigo/lib/python2.7/dist-packages/tornado/__init__.pyc
alexv@AlexV-Linux:~$ pip freeze | grep tornado
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
tornado==4.2.1

But it could also be related to the fact that I repackaged tornado (4.2.1) to be installed on its own, not sneakily as part of rosbridge-server package (tornado v4.0.2)....

To test that theory I reinstall the old system pip on my trusty distro : 

alexv@AlexV-Linux:~$ sudo pip uninstall pip
[sudo] password for alexv: 
The directory '/home/alexv/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Uninstalling pip-8.1.2:
[...]
Proceed (y/n)? y
  Successfully uninstalled pip-8.1.2
The directory '/home/alexv/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
You are using pip version 1.5.4, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
alexv@AlexV-Linux:~$ pip --version
The program 'pip' is currently not installed. You can install it by typing:
sudo apt-get install python-pip
alexv@AlexV-Linux:~$ sudo apt-get install python-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python-pip is already the newest version.
The following packages were automatically installed and are no longer required:
  firefox-locale-ko libhdb9-heimdal libkdc2-heimdal libntdb1
  linux-headers-3.13.0-65 linux-headers-3.13.0-65-generic
  linux-image-3.13.0-65-generic linux-image-extra-3.13.0-65-generic
  python-ntdb
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.
alexv@AlexV-Linux:~$ pip list | grep ros
The program 'pip' is currently not installed. You can install it by typing:
sudo apt-get install python-pip
alexv@AlexV-Linux:~$ sudo apt-get install python-pip --reinstall
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  firefox-locale-ko libhdb9-heimdal libkdc2-heimdal libntdb1
  linux-headers-3.13.0-65 linux-headers-3.13.0-65-generic
  linux-image-3.13.0-65-generic linux-image-extra-3.13.0-65-generic
  python-ntdb
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 4 not upgraded.
Need to get 97.3 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://kr.archive.ubuntu.com/ubuntu/ trusty-updates/universe python-pip all 1.5.4-1ubuntu4 [97.3 kB]
Fetched 97.3 kB in 0s (400 kB/s)    
(Reading database ... 1216874 files and directories currently installed.)
Preparing to unpack .../python-pip_1.5.4-1ubuntu4_all.deb ...
Unpacking python-pip (1.5.4-1ubuntu4) over (1.5.4-1ubuntu4) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up python-pip (1.5.4-1ubuntu4) ...

I also had to reinstall the system tornado : 

alexv@AlexV-Linux:~$ python -c "import tornado; print tornado.__file__; print tornado.version"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named tornado
alexv@AlexV-Linux:~$ /usr/lib/python2.7/dist-packages/tornado/__init__.pyc
bash: /usr/lib/python2.7/dist-packages/tornado/__init__.pyc: No such file or directory
alexv@AlexV-Linux:~$ sudo apt-get install python-tornado
[sudo] password for alexv: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  firefox-locale-ko libhdb9-heimdal libkdc2-heimdal libntdb1
  linux-headers-3.13.0-65 linux-headers-3.13.0-65-generic
  linux-image-3.13.0-65-generic linux-image-extra-3.13.0-65-generic
  python-ntdb
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  python-mysqldb
Suggested packages:
  python-egenix-mxdatetime mysql-server-5.1 mysql-server python-mysqldb-dbg
The following NEW packages will be installed:
  python-mysqldb python-tornado
0 upgraded, 2 newly installed, 0 to remove and 4 not upgraded.
Need to get 270 kB of archives.
After this operation, 1,473 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://kr.archive.ubuntu.com/ubuntu/ trusty/main python-mysqldb amd64 1.2.3-2ubuntu1 [55.4 kB]
Get:2 http://kr.archive.ubuntu.com/ubuntu/ trusty/main python-tornado all 3.1.1-1ubuntu2 [215 kB]
Fetched 270 kB in 0s (698 kB/s)         
Selecting previously unselected package python-mysqldb.
(Reading database ... 1216874 files and directories currently installed.)
Preparing to unpack .../python-mysqldb_1.2.3-2ubuntu1_amd64.deb ...
Unpacking python-mysqldb (1.2.3-2ubuntu1) ...
Selecting previously unselected package python-tornado.
Preparing to unpack .../python-tornado_3.1.1-1ubuntu2_all.deb ...
Unpacking python-tornado (3.1.1-1ubuntu2) ...
Setting up python-mysqldb (1.2.3-2ubuntu1) ...
Setting up python-tornado (3.1.1-1ubuntu2) ...

And repeating the same test as before : 

alexv@AlexV-Linux:~$ python -c "import tornado; print tornado.__file__; print tornado.version"
/usr/lib/python2.7/dist-packages/tornado/__init__.pyc
3.1.1
alexv@AlexV-Linux:~$ pip freeze | grep tornado
tornado==3.1.1
alexv@AlexV-Linux:~$ source /opt/ros/indigo/setup.bash 
alexv@AlexV-Linux:~$ python -c "import tornado; print tornado.__file__; print tornado.version"
/opt/ros/indigo/lib/python2.7/dist-packages/tornado/__init__.pyc
4.2.1
alexv@AlexV-Linux:~$ pip freeze | grep tornado
tornado==4.2.1

So it seems this is issue arises when a python package (tornado) is installed as a part of another python package (rosbridge-server) instead of on its own (tornado) ???

To verify, I uninstalled ros-indigo-tornado and reinstalled ros-indigo-rosbridge-server : 

alexv@AlexV-Linux:~$ sudo apt-get remove ros-indigo-tornado 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  firefox-locale-ko libhdb9-heimdal libkdc2-heimdal libntdb1
  linux-headers-3.13.0-65 linux-headers-3.13.0-65-generic
  linux-image-3.13.0-65-generic linux-image-extra-3.13.0-65-generic
  python-ntdb python-singledispatch ros-indigo-backports-ssl-match-hostname
  ros-indigo-certifi
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  ros-indigo-tornado
0 upgraded, 0 newly installed, 1 to remove and 4 not upgraded.
After this operation, 2,998 kB disk space will be freed.
Do you want to continue? [Y/n] 
(Reading database ... 1217063 files and directories currently installed.)
Removing ros-indigo-tornado (4.2.1-3trusty-20170323-091448-0700) ...
alexv@AlexV-Linux:~$ sudo apt-get install ros-indigo-rosbridge-
ros-indigo-rosbridge-library  ros-indigo-rosbridge-server   ros-indigo-rosbridge-suite
alexv@AlexV-Linux:~$ sudo apt-get install ros-indigo-rosbridge-server 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  firefox-locale-ko libhdb9-heimdal libkdc2-heimdal libntdb1
  linux-headers-3.13.0-65 linux-headers-3.13.0-65-generic
  linux-image-3.13.0-65-generic linux-image-extra-3.13.0-65-generic
  python-ntdb python-singledispatch ros-indigo-certifi
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  python-bson python-bson-ext ros-indigo-rosapi ros-indigo-rosauth
  ros-indigo-rosbridge-library
The following packages will be REMOVED:
  ros-indigo-backports-ssl-match-hostname
The following NEW packages will be installed:
  python-bson python-bson-ext ros-indigo-rosapi ros-indigo-rosauth
  ros-indigo-rosbridge-library ros-indigo-rosbridge-server
0 upgraded, 6 newly installed, 1 to remove and 4 not upgraded.
Need to get 515 kB of archives.
After this operation, 4,167 kB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://kr.archive.ubuntu.com/ubuntu/ trusty/main python-bson amd64 2.6.3-1build1 [18.4 kB]
Get:2 http://kr.archive.ubuntu.com/ubuntu/ trusty/main python-bson-ext amd64 2.6.3-1build1 [17.5 kB]
Get:3 http://packages.ros.org/ros-shadow-fixed/ubuntu/ trusty/main ros-indigo-rosbridge-library amd64 0.7.16-0trusty-20170313-095924-0700 [88.7 kB]
Get:4 http://packages.ros.org/ros-shadow-fixed/ubuntu/ trusty/main ros-indigo-rosapi amd64 0.7.16-0trusty-20170313-104233-0700 [64.6 kB]
Get:5 http://packages.ros.org/ros-shadow-fixed/ubuntu/ trusty/main ros-indigo-rosauth amd64 0.1.7-0trusty-20170313-093021-0700 [29.1 kB]
Get:6 http://packages.ros.org/ros-shadow-fixed/ubuntu/ trusty/main ros-indigo-rosbridge-server amd64 0.7.16-0trusty-20170313-104800-0700 [297 kB]
Fetched 515 kB in 3s (146 kB/s)                      
(Reading database ... 1216874 files and directories currently installed.)
Removing ros-indigo-backports-ssl-match-hostname (3.5.0-4trusty-20170323-090449-0700) ...
Selecting previously unselected package python-bson.
(Reading database ... 1216856 files and directories currently installed.)
Preparing to unpack .../python-bson_2.6.3-1build1_amd64.deb ...
Unpacking python-bson (2.6.3-1build1) ...
Selecting previously unselected package python-bson-ext.
Preparing to unpack .../python-bson-ext_2.6.3-1build1_amd64.deb ...
Unpacking python-bson-ext (2.6.3-1build1) ...
Selecting previously unselected package ros-indigo-rosbridge-library.
Preparing to unpack .../ros-indigo-rosbridge-library_0.7.16-0trusty-20170313-095924-0700_amd64.deb ...
Unpacking ros-indigo-rosbridge-library (0.7.16-0trusty-20170313-095924-0700) ...
Selecting previously unselected package ros-indigo-rosapi.
Preparing to unpack .../ros-indigo-rosapi_0.7.16-0trusty-20170313-104233-0700_amd64.deb ...
Unpacking ros-indigo-rosapi (0.7.16-0trusty-20170313-104233-0700) ...
Selecting previously unselected package ros-indigo-rosauth.
Preparing to unpack .../ros-indigo-rosauth_0.1.7-0trusty-20170313-093021-0700_amd64.deb ...
Unpacking ros-indigo-rosauth (0.1.7-0trusty-20170313-093021-0700) ...
Selecting previously unselected package ros-indigo-rosbridge-server.
Preparing to unpack .../ros-indigo-rosbridge-server_0.7.16-0trusty-20170313-104800-0700_amd64.deb ...
Unpacking ros-indigo-rosbridge-server (0.7.16-0trusty-20170313-104800-0700) ...
Setting up python-bson (2.6.3-1build1) ...
Setting up python-bson-ext (2.6.3-1build1) ...
Setting up ros-indigo-rosbridge-library (0.7.16-0trusty-20170313-095924-0700) ...
Setting up ros-indigo-rosapi (0.7.16-0trusty-20170313-104233-0700) ...
Setting up ros-indigo-rosauth (0.1.7-0trusty-20170313-093021-0700) ...
Setting up ros-indigo-rosbridge-server (0.7.16-0trusty-20170313-104800-0700) ...
alexv@AlexV-Linux:~$ python -c "import tornado; print tornado.__file__; print tornado.version"
/opt/ros/indigo/lib/python2.7/dist-packages/tornado/__init__.pyc
4.0.2
alexv@AlexV-Linux:~$ pip freeze | grep tornado
tornado==3.1.1

And the problem is here again.

So apparently there is some ROS packaging limitation when used with multiple python packages inside one ROS package ? Something that should not be abused and we should prevent against ? How can we confirm that ??

In anycase it doesn't seem related to rosdep, so we should find the root cause and recreate an issue in the appropriate repo.

wjwwood commented 7 years ago

Since this doesn't appear to be related to how rosdep uses pip, I'll close this for now. It can be reopened if need be.