Python issues with Foxy and Ubuntu Focal

[mr337]

I am having packaging issues when package.xml has a pip requirement, can't install via system packages.

The error I am getting is:

mr337@ubuntu /tmp/blah $ colcon bundle                                                                                                                                                                               
Bundling workspace...                                                                                    
Cache is not valid. Clearing cache...                                                                                                                                                                              
Reading package lists... Done                                                                            
Building dependency tree... Done                                                                                                                                                                                   
Checking if dependency tarball exists...                                                                 
Starting >>> project                                                                                 
loading view [*default*] with rospkg loader                                                              
--- stderr: project                                                                                  
Traceback (most recent call last):                                                                       
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)                                                                                                                                                                          
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_core/task/__init__.py", line 93, in __call__
    return await task_method(*args, **kwargs)                                                            
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_ros_bundle/task/ament_python/bundle.py", line 28, in bundle
    await super().bundle()                                                                                                                                                                                         
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_ros_bundle/task/ros_bundle.py", line 67, in bundle
    args.installers['apt'].add_to_install_list('python-pip')                                                                                                                                                       
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_bundle/installer/apt.py", line 181, in add_to_install_list
    if not self.is_package_available(package_key):                                                                                                                                                                 
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_bundle/installer/apt.py", line 176, in is_package_available
    return self._cache[package_key] is not None                                                                                                                                                                    
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 305, in __getitem__                                                                                                                                     
    raise KeyError('The cache has no package named %r' % key)                                                                                                                                                      
KeyError: "The cache has no package named 'python-pip'"
---                                                                                                                                                                                                                
Failed   <<< project [0.06s, exited with code 1]

After doing some digging it looks like the ros-bundle colcon plugin is adding the python2 version of pip which doesn't exist for Focal. Only the system packages python3-pip is available.

It looks like the ros-bundle is blindly injecting python-pip https://github.com/colcon/colcon-ros-bundle/blob/master/colcon_ros_bundle/task/ros_bundle.py#L67.

Does anyone have any objections to moving that to python3-pip since all the current ROS distros are python3 compatible or only support python3? Happy to make a PR.

[mr337]

Whoops didn't see that there is a related ticket for Foxy support :(

https://github.com/colcon/colcon-ros-bundle/issues/40

[mr337]

Also making a quick one line change to use python3-pip resulted in another error further down the process.

Fetched 86.5 MB in 6s (0 B/s)                                                                                                                                                                                     
Extracting apt packages...
[259.729s] ERROR:colcon:colcon bundle: [Errno 2] No such file or directory: '/tmp/blah/bundle/cache/bundle_staging/usr/bin/python2'
Traceback (most recent call last):
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_core/command.py", line 528, in verb_main
    rc = context.args.main(context=context)
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_bundle/verb/bundle.py", line 135, in main
    dependencies_changed = self._manage_dependencies(
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_bundle/verb/bundle.py", line 186, in _manage_dependencies
    self._installer_manager.run_installers(
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_bundle/verb/_installer_manager.py", line 78, in run_installers
    installer_metadata[name] = installer.install()
  File "/home/mr337/.local/lib/python3.8/site-packages/colcon_bundle/installer/base_pip_installer.py", line 86, in install
    subprocess.check_call(
  File "/usr/lib/python3.8/subprocess.py", line 359, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 340, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/blah/bundle/cache/bundle_staging/usr/bin/python2'

[mr337]

So I have a quick and dirty fix for this, changing line https://github.com/colcon/colcon-bundle/blob/master/colcon_bundle/installer/pip.py#L28 to python3 will take care of the issue.

Why is this dirty? It seems there is a https://github.com/colcon/colcon-bundle/blob/master/colcon_bundle/installer/pip3.py file that is supposed to be used instead. I don't know why that isn't being properly called. I do see that colcon bundle has pip3 arguments but they don't seem to align with reading out of the package.xml file.

$ colcon bundle --help
.
.
.
Arguments for 'pip3' installer:
  --pip3-args [* [* ...]]
                        Pass arguments to CMake projects. Arguments matching
                        other options in colcon must be prefixed by a space,
                        e.g. --pip3-args " --help"
  --pip3-requirements PIP3_REQUIREMENTS
                        Path to a requirements.txt. All packages in the
                        filewill be installed into Python3 in the bundle

[mr337]

I think this maybe related, sigh, https://github.com/colcon/colcon-bundle/issues/171

[mr337]

I have created to PRs that address two different issues. Until I can get them merged the quick fix is to install via pip the PRs.

$ pip3 install git+https://github.com/mr337/colcon-ros-bundle.git@python3_pip_upgrade
$ pip3 install git+https://github.com/mr337/colcon-bundle.git@drop_python2_support

[Niknu]

@cottsay, do you have any idea why this isn't done? I'm worried that bundle will be deprecated in the future as ROS2 is moving forward and the OS system also changes

[cottsay]

cottsay, do you have any idea why this isn't done?

I don't believe anyone is actively maintaining this package. I've made some housekeeping changes to it, but the last functional change to the package was over two years ago, and the documentation still states that the latest supported Ubuntu release is Bionic.

Anyone from @colcon/aws-robotics care to comment on their plans here?

[mr337]

I don't speak for aws-robotics, but was a user of this packaging method using AWS. At the time AWS Robomaker recommending packaging into a bundle for deployment. Since then AWS Robomaker has deprecated deployment in favor of AWS GreenGrass with deployments [1].

With AWS GreenGrass the docs seem to indicate Docker support instead [2]. So just as a guess I don't think there is effort within aws-robotsics to support this anymore.

1 - https://aws.amazon.com/blogs/robotics/deploy-robotic-applications-using-aws-robomaker/ 2 - https://aws.amazon.com/blogs/robotics/deploy-and-manage-ros-robots-with-aws-iot-greengrass-2-0-and-docker/

[emersonknapp]

I'm no longer at aws-robotics, so can't speak for them, but as far as I know there is nobody there currently planning to maintain this tool. Like @mr337 mentioned, I think the priority shifted and this tool was deprecated.