search

ros2 / ros2

The Robot Operating System, is a meta operating system for robots.
https://docs.ros.org
3.65k stars 683 forks source link

Using Python Virtual Environments #1094

Open jandaa opened 3 years ago

jandaa commented 3 years ago

Bug report

I'm trying to use a python virtual environment in my colcon workspace to separate system level and node level dependencies. I'm following the instructions given in this tutorial in the ros2 documentation (Specifically installing via a virtual environment). However, when I check what python interpreter is being used inside the node, it returns the system level one instead of the virtual environment. This can be further verified by installing a package in the virtual environment that is not present in the system environment and try importing it from inside the node.

I have tried doing a clean build with and without sourcing the virtual environment in the build shell. I have also tried using virtualenv as in the tutorial and also python -m venv. I also call my virtual environment "venv" as in the tutorial.

Required Info:

Steps to reproduce issue

  1. Follow the steps in the tutorial
  2. Create a ros2 python package with cd src && ros2 pkg create venv --build-type ament_python
  3. Create a python node called venv.py under src/venv/venv with the following code 
    
import rclpy
from rclpy.node import Node
import sys

class ExampleNode(Node): def init(self): super().init('sensors') self.get_logger().info("ExampleNode started") self.get_logger().info(f"python path: {sys.executable}")

def main(args=None): rclpy.init(args=args) node = ExampleNode() rclpy.spin(node) rclpy.shutdown()

if name == 'main': main()

4. Add the node to setup.py with

entry_points={ 'console_scripts': [ 'venv = venv.venv:main' ], },

5. Build using

source ./venv/bin/activate source /opt/ros/foxy/setup.bash colcon build source install/setup.sh


6. Run using ```ros2 run venv venv ```

#### Expected behavior
The returned python path should show ```${workspaceFolder}/venv/bin/python``` and any packages installed in the virtual environment should be accessible inside the node. 

#### Actual behavior
Returns the system base python ```/usr/bin/python3``` and breaks when trying to import a package that is installed in the virtual environment but not at system level.
YunfeiZHAO commented 3 years ago

same problem with anaconda

PymZoR commented 3 years ago

Same problem on galactic

clalancette commented 3 years ago

In short, python venvs with Python installed in the venv are never going to work using the installed ROS 2 binaries. The problem boils down to the fact that building a Python package as a Debian hard-codes the default python path in the console script (namely /usr/bin/python3).

However, what should work is having a Python venv with just pip packages installed. That is, if you rely on the system version of Python, but just make a venv where you pip install a bunch of dependencies, that should work just fine. That said, ROS 2 expects certain versions of pip dependencies/APIs to be present, so it may be the case that you cannot use newer versions of pip packages while doing this.

jandaa commented 3 years ago

Thank you so much @clalancette for your response, that makes sense! What you describe as using just the pip packages would work just fine. In this case, how would we point our environment to these separate site packages contained in our venv?

clalancette commented 3 years ago

Thank you so much @clalancette for your response, that makes sense! What you describe as using just the pip packages would work just fine. In this case, how would we point our environment to these separate site packages contained in our venv?

Just running source venv/bin/activate, then source /opt/ros/foxy/setup.bash should do it, though I haven't personally tried this out yet.

theunkn0wn1 commented 3 years ago

Thank you so much @clalancette for your response, that makes sense! What you describe as using just the pip packages would work just fine. In this case, how would we point our environment to these separate site packages contained in our venv?

Just running source venv/bin/activate, then source /opt/ros/foxy/setup.bash should do it, though I haven't personally tried this out yet.

I tried doing precisely this, though ros2 run still seems to be using either wrong interpreter or is otherwise disregarding venv packges and thus isn't finding the pip-installed packages the node requires.

Console output provided. console.log

clalancette commented 3 years ago

It's possible that sourcing install/setup.zsh is overwriting the PYTHONPATH that the venv setup. Since you don't actually need the venv to build, then I'll suggest trying the other order:

theunkn0wn1 commented 3 years ago

It's possible that sourcing install/setup.zsh is overwriting the PYTHONPATH that the venv setup. Since you don't actually need the venv to build, then I'll suggest trying the other order:

* source /opt/ros/foxy/setup.zsh

* colcon build

* source install/local_setup.zsh

* source venv/bin/activate

* ros2 run pkg exe
~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 12:57:43
$ source /opt/ros/foxy/setup.zsh

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 12:57:51
$ colcon build
Starting >>> osiris_turret
Finished <<< osiris_turret [0.50s]          

Summary: 1 package finished [0.66s]

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 12:57:54
$ source install/local_setup.zsh

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 12:57:56
$ source venv/bin/activate
(venv) 
~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 12:57:59
$ ros2 run osiris_turret service
Traceback (most recent call last):
  File "/home/orion/projects/skool/rover/next/science/osiris_turret_ros2/install/osiris_turret/lib/osiris_turret/service", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3254, in <module>
    def _initialize_master_working_set():
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3237, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3266, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 584, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 901, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 787, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'turret_python_interface' distribution was not found and is required by osiris-turret
(venv) 
~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 12:58:02
$
aprotyas commented 3 years ago

@theunkn0wn1 can you do a printenv and check what PYTHONPATH is set to?

theunkn0wn1 commented 3 years ago

@aprotyas it seems venv isn't adding anything to the PYTHONPATH, but rather extending PATH and setting the VIRTUAL_ENV key.

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:16:50
$ printenv  | grep -i venv

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:16:54
$ printenv  | grep -i pythonpath

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:16:59
$ source /opt/ros/foxy/setup.zsh

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:17:03
$ printenv  | grep -i pythonpath
PYTHONPATH=/opt/ros/foxy/lib/python3.8/site-packages

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:17:06
$ printenv  | grep -i venv      

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:17:14
$ source ./install/local_setup.zsh

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:17:21
$ printenv  | grep -i venv

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:17:24
$ printenv  | grep -i pythonpath  
PYTHONPATH=/home/orion/projects/skool/rover/next/science/osiris_turret_ros2/install/osiris_turret/lib/python3.8/site-packages:/opt/ros/foxy/lib/python3.8/site-packages

~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:17:26
$ source venv/bin/activate
(venv) 
~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:17:31
$ printenv  | grep -i pythonpath
PYTHONPATH=/home/orion/projects/skool/rover/next/science/osiris_turret_ros2/install/osiris_turret/lib/python3.8/site-packages:/opt/ros/foxy/lib/python3.8/site-packages
(venv) 
~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:17:34
$ printenv  | grep -i venv      
PATH=/home/orion/projects/skool/rover/next/science/osiris_turret_ros2/venv/bin:/opt/ros/foxy/bin:/home/orion/gems/bin:/home/orion/.cargo/bin:/home/orion/.cargo/bin:/home/orion/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
VIRTUAL_ENV=/home/orion/projects/skool/rover/next/science/osiris_turret_ros2/venv
PS1=(venv) 
(venv) 
~/projects/skool/rover/next/science/osiris_turret_ros2 ⌚ 13:17:37
$
clalancette commented 3 years ago

OK, I see what is happening here. The issue is in the script that python generates when we do the install. For instance, in the original case in this PR, the build process ends up generating install/venv/lib/venv/venv. Inside of that script, the preamble looks something like:

#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'venv==0.0.0','console_scripts','venv'
import re
import sys

Because that is a hard-coded path to /usr/bin/python3, it isn't even looking inside of the virtual environment for packages. What that really should be is #!/usr/bin/env python3 so that it would pick up the venv properly.

I'm really not sure how we can fix that; that script is totally generated by Python itself. We'll have to look into whether there is a way to have EASY-INSTALL-ENTRY-SCRIPT honor the venv, in a cross-platform way.

theunkn0wn1 commented 3 years ago

One possible solution would be for ros2 run to detect if the specified executable is a python target, and run the entry-point using the environment's interpreter directly instead of depending on the shebang line. While this is a workaround, it would resolve the immediate issue.

I am not entirely familiar with how ROS2's py build process works, but if EASY-INSTALL-ENTRY-SCRIPT is referring to easy-install, that is depricated.

Another possible option I am considering is running a post-processing script over the generated entry-points to programmatically replace the generated shebang lines.

theunkn0wn1 commented 3 years ago

Caused by https://github.com/colcon/colcon-core/issues/181 Fixed by, but not merged, https://github.com/colcon/colcon-core/pull/183. I tried the fix in https://github.com/colcon/colcon-core/pull/188 but that seems to not effect the core issue. I applied the fix in 183 against my local ROS2 python package's setup.cfg and it fixes the shebang bug.

yliu0062 commented 3 years ago

I think I also encounter this issue, after running this "sudo apt install ros-foxy-desktop", the pythonpath will have additional path "opt/ros/foxy/lib/python3.8/site-packages", then the right virtual python env can't be recognized correctly.

theunkn0wn1 commented 3 years ago

Here are some steps to creating a suitable venv:

  1. source your ros distro's setup
  2. create the venv, enabling system packages by symlink. python3 -m venv venv --system-site-packages --symlinks
  3. source the venv's setup script

In terms of creating Python ROS2 packages, create it as normal but add apply the fix in https://github.com/colcon/colcon-core/pull/183 to your setup.cfg

When building your nodes, be sure you've sourced both ros's setup as well as the venv's, BEFORE invoking colcon build.

miRemid commented 2 years ago

thanks to @theunkn0wn1 , it works

devrite commented 2 years ago

Since we are working with (potentially single vens). Instead of adding the "fixed" version, colcon already extracts the correct interpreter if built from your workspace or when it is reinstalled in your venv using something like

pip install --force-reinstall colcon-core setuptools==$(pip list --no-index --format=json | jq -r '.[] | select(.name=="setuptools").version')

Of course you can not auto maintain multiple installations along side it (single colcon install). You need it for every venv. Which is a reasonable compromise, I guess. The above setuptools limit is only needed if an upgraded version causes issues. For example the current (py)torch still accesses features of distutils at runtime which have been removed in newer versions. But pip auto upgrades it for some reason although it is not required. Only-if-needed does still install a newer version.

When colcon core is being rebuilt it will your sys.executable path and add it to your python scripts as shebang. In case of a venv or any python env this is the interpreter path of your executable. No need to manually set the PYTHONPATH or using the specific fix. If you want to work with a single install of colcon though this may be the version to go.

mshahbionaut commented 2 years ago

This thread was super helpful. I couldn't get @theunkn0wn1 's solution to work for me, but I might be doing something different (like using virtualenv instead of venv, not sure how/why that would be different)

My application is simple for now, and I can just rely on the system python and install my packages there. However, is there a long term cleaner solution even possible for this? Could we try to update documentation here to reflect the current behavior and/or the fix?

Thanks!

EDIT:

@theunkn0wn1 's solution does work with virtualenv too... After an hour+ of trying other things, upon trying this I forgot to actually install the package at all in my virtualenv xD

However, I'm still at the mercy of including the change in setup.cfg for all my packages and the order in which I source setup files.

jdlangs commented 1 year ago

Would this also be fixed by addressing colcon using the deprecated method of installing by directly invoking setup.py?

jdlangs commented 1 year ago

Just to follow up on this, I found everything works well if I just run python -m colcon instead of colcon when the virtualenv is active. No need to change any files in your packages.

xiaolong73 commented 1 year ago

I found a way that you can use virtual python's package like torch.

import sys
sys.path.append('{YOUR VIRTUAL PYTHON PATH}/lib/python3.10/site-packages')
import torch
theunkn0wn1 commented 1 year ago

I found a way that you can use virtual python's package like torch.

import sys
sys.path.append('{YOUR VIRTUAL PYTHON PATH}/lib/python3.10/site-packages')
import torch

Note: path hacks make the code very brittle, break isolation, and make environments difficult to reproduce. I encourage you to find a better solution.

Ryanf55 commented 1 year ago

2. python3 -m venv venv --system-site-packages --symlinks

This doesn't work, still can't find lark, despite it being installed.

--- stderr: nav2_msgs                                                                                                                                                                                                                       
Traceback (most recent call last):                                                                                                                                                                                                          
  File "/home/ryan/Development/ros2_rolling/install/rosidl_generator_type_description/lib/rosidl_generator_type_description/rosidl_generator_type_description", line 21, in <module>                                                        
    from rosidl_generator_type_description import generate_type_hash
  File "/home/ryan/Development/ros2_rolling/install/rosidl_generator_type_description/lib/python3.11/site-packages/rosidl_generator_type_description/__init__.py", line 24, in <module>
    from rosidl_parser.parser import parse_idl_file
  File "/home/ryan/Development/ros2_rolling/install/rosidl_parser/lib/python3.11/site-packages/rosidl_parser/parser.py", line 20, in <module>
    from lark import Lark
(.rolling) ryan@ryan-B650-970:~/Development/nav2_ws/src/navigation2$ python3 -m pip list | grep lark
lark                                 1.1.5

python -m colcon

This doesn't work because:

$ which colcon
/usr/bin/colcon
devrite commented 1 year ago
  1. python3 -m venv venv --system-site-packages --symlinks

This doesn't work, still can't find lark, despite it being installed.

--- stderr: nav2_msgs                                                                                                                                                                                                                       
Traceback (most recent call last):                                                                                                                                                                                                          
  File "/home/ryan/Development/ros2_rolling/install/rosidl_generator_type_description/lib/rosidl_generator_type_description/rosidl_generator_type_description", line 21, in <module>                                                        
    from rosidl_generator_type_description import generate_type_hash
  File "/home/ryan/Development/ros2_rolling/install/rosidl_generator_type_description/lib/python3.11/site-packages/rosidl_generator_type_description/__init__.py", line 24, in <module>
    from rosidl_parser.parser import parse_idl_file
  File "/home/ryan/Development/ros2_rolling/install/rosidl_parser/lib/python3.11/site-packages/rosidl_parser/parser.py", line 20, in <module>
    from lark import Lark
(.rolling) ryan@ryan-B650-970:~/Development/nav2_ws/src/navigation2$ python3 -m pip list | grep lark
lark                                 1.1.5

python -m colcon

This doesn't work because:

$ which colcon
/usr/bin/colcon

Have you tried reinstalling colcon-core to your env?

Ryanf55 commented 1 year ago
  1. python3 -m venv venv --system-site-packages --symlinks

This doesn't work, still can't find lark, despite it being installed.

--- stderr: nav2_msgs                                                                                                                                                                                                                       
Traceback (most recent call last):                                                                                                                                                                                                          
  File "/home/ryan/Development/ros2_rolling/install/rosidl_generator_type_description/lib/rosidl_generator_type_description/rosidl_generator_type_description", line 21, in <module>                                                        
    from rosidl_generator_type_description import generate_type_hash
  File "/home/ryan/Development/ros2_rolling/install/rosidl_generator_type_description/lib/python3.11/site-packages/rosidl_generator_type_description/__init__.py", line 24, in <module>
    from rosidl_parser.parser import parse_idl_file
  File "/home/ryan/Development/ros2_rolling/install/rosidl_parser/lib/python3.11/site-packages/rosidl_parser/parser.py", line 20, in <module>
    from lark import Lark
(.rolling) ryan@ryan-B650-970:~/Development/nav2_ws/src/navigation2$ python3 -m pip list | grep lark
lark                                 1.1.5

python -m colcon

This doesn't work because:

$ which colcon
/usr/bin/colcon

Have you tried reinstalling colcon-core to your env?

I've removed the use of colcon through apt and will now try through python instead.

Here's the latest progress of using venv with Ubuntu 23 and ROS 2 humble: https://github.com/Ryanf55/ros-on-lunar/blob/add-lunar-dockerfile/Dockerfile

TZECHIN6 commented 1 year ago

So... is it recommended to use venv in ros2 development? ros2 document shows a way to use venv, but simply its not working....

jdlangs commented 1 year ago

python -m colcon

This doesn't work because:

$ which colcon
/usr/bin/colcon

The purpose of using python -m colcon is exactly that it doesn't invoke /usr/bin/colcon. If you run colcon explicitly through your virtual env interpreter version then setuptools will use that one inside the generated console scripts instead of /usr/bin/python3.

I'm not sure if it was mentioned above, but it also works to pip install colcon-common-extensions into your virtualenv and make sure which colcon points to that one. In both cases, the main limitation is that you have to have the virtualenv active when you build the workspace.

123swk123 commented 1 year ago

I have been facing same issue while building ros2 examples on pyenv virtual environment (colcon installed to pyenv pip) & ROS2 nightly build.

Applied the following work around.

jjdengjj commented 7 months ago

Try this solution:

  1. Modify "setup.cfg" in your ros2 python module as follows: 
    [build_scripts]
executable = /usr/bin/env python3
  2. Rebuild your ros2 package, and then source the setup.bash in the terminal.
  3. Activate the corresponding virtual environment by source the activate file.
  4. Run your module with ros2 run ...., which will be able to identify the right virtual python environment and installed package in it.
hugoyvrn commented 7 months ago

@123swk123 I think the issue is due to the --symlin-install option.

@jjdengjj It works for me without the --symlin-install option. But not with the --symlin-install option. Do you have a clue?

jjdengjj commented 7 months ago

@hugoyvrn I didn't use --symlink-install flag. I believe the key issue is that without specify the executable as /usr/bin/env python3 in setup.cfg, the ros2 run will use the default python in the os system, which can only find the system wise package (including those ones installed with rosdep).

So far, my solution works fine on ros2 humble, and it is easy to debug those ros python codes in virtual environment with vscode.

hugoyvrn commented 7 months ago

Thank you @jjdengjj for your quick reply. I understood what you said, and it works fine for me.

But I do want to use the --symlink-install flag. When I do, my python node runs on the default python, not in the virtual env as I expect.

Hawxxer commented 5 months ago

Sorry but how can such a fundamental issue be 3 years old? Is this not applicant for all users, do we overlook sth? Are majority just install their packages on the plain machine? Are they just using rosdep?

dghw commented 3 months ago

I think the 'Installing via a virtual environment' section of https://docs.ros.org/en/jazzy/How-To-Guides/Using-Python-Packages.html should be removed or point to this issue to prevent wasted effort.

I am confused on best practices now to integrate an external/custom Python package with ROS2 code, specifically one that is not installable via package manager (apt/pip).

Is the only way forward to strip the package in to a module within a ROS2 node and maintain the standalone package/module version separately...? Any tips from others who have met this issue?

alaurenzi commented 2 months ago

Dear maintainers, is there any plan to address this issue?

With Ubuntu24 + Jazzy (and therefore Python 3.12) it is not possible to pip install software outside of a virtualenv without "breaking system packages" as you know. This makes solving this issue pretty critical. Or am I missing something?

clalancette commented 2 months ago

is there any plan to address this issue?

Yes and no. Yes, there is an open set of PRs that I need to get back to that will improve the situation.

However, it will probably not be possible to solve this in the general case. In particular, if you have a custom version of Python and/or pip packages installed that are compiled in a venv, that is pretty much fundamentally incompatible with ROS 2 binaries. It may possible to make that scenario work in a pure from-source build, which I'll look into when I open the documentation PR that corresponds to https://github.com/ros2/ros2/pull/1524

victorcarvesk commented 4 weeks ago

Dear maintainers, is there any plan to address this issue?

With Ubuntu24 + Jazzy (and therefore Python 3.12) it is not possible to pip install software outside of a virtualenv without "breaking system packages" as you know. This makes solving this issue pretty critical. Or am I missing something?

I’m currently facing this same issue (with the same setup), and all the proposed solutions seem to be more of a workaround than a robust solution. Why is it not possible to just specify the shebang in the Python scripts inside an ament_python package and be done with it? I tried this approach, but the shebang seems to be ignored in ament_python packages.

Using Python scripts in ament_cmake packages with shebangs (#!/usr/bin/env python3) inside virtual environments works seamlessly. However, in theory, ament_python packages should be better suited for handling pure Python packages.

Considering that all Linux systems (and possibly other systems) have an environment variable pointing to a Python interpreter, it's strange that ROS simply ignores it and uses a predefined interpreter.