Pip version of colcon unable to build ROS2 on Jammy

[mjcarroll]

There is an issue with the pip3 installed version of colcon which causes it to be unable to build ROS2 rolling on ubuntu jammy.

I have narrowed the issue to failing on ament_cmake_core with a particular set of colcon versions.

An example with a dockerfile that fails

https://gist.github.com/mjcarroll/a63c422fa8578a0999b5d8b20be76c96#file-dockerfile-bad

An example of a dockerfile pinned to specific colcon versions that works:

https://gist.github.com/mjcarroll/a63c422fa8578a0999b5d8b20be76c96#file-dockerfile-good

To test:

docker build -t ros-rolling:jammy .
docker run -it ros-rolling:jammy /bin/bash
cd ~/ros2_rolling
colcon build --packages-up-to ament_cmake_core

Error output:

--- stderr: ament_cmake_core                         
Traceback (most recent call last):
  File "/root/ros2_rolling/src/ament/ament_cmake/ament_cmake_core/cmake/package_templates/templates_2_cmake.py", line 21, in <module>
    from ament_package.templates import get_environment_hook_template_path
ModuleNotFoundError: No module named 'ament_package'
CMake Error at ament_cmake_package_templates-extras.cmake:41 (message):
  execute_process(/usr/bin/python3.10
  /root/ros2_rolling/src/ament/ament_cmake/ament_cmake_core/cmake/package_templates/templates_2_cmake.py
  /root/ros2_rolling/build/ament_cmake_core/ament_cmake_package_templates/templates.cmake)
  returned error code 1
Call Stack (most recent call first):
  CMakeLists.txt:19 (include)

---
Failed   <<< ament_cmake_core [0.22s, exited with code 1]

$ colcon version-check
colcon-argcomplete 0.3.3: up-to-date
colcon-bash 0.4.2: up-to-date
colcon-cd 0.1.1: up-to-date
colcon-cmake 0.2.26: up-to-date
colcon-core 0.8.1: up-to-date
colcon-defaults 0.2.6: up-to-date
colcon-devtools 0.2.3: up-to-date
colcon-library-path 0.2.1: up-to-date
colcon-metadata 0.2.5: up-to-date
colcon-notification 0.2.13: up-to-date
colcon-output 0.2.12: up-to-date
colcon-package-information 0.3.3: up-to-date
colcon-package-selection 0.2.10: up-to-date
colcon-parallel-executor 0.2.4: up-to-date
colcon-pkg-config 0.1.0: up-to-date
colcon-powershell 0.3.7: up-to-date
colcon-python-setup-py 0.2.7: up-to-date
colcon-recursive-crawl 0.2.1: up-to-date
colcon-ros 0.3.23: up-to-date
colcon-test-result 0.3.8: up-to-date
colcon-zsh 0.4.0: up-to-date

[nuclearsandwich]

For later triangulation, here's a ROS 2 CI job which is building ament_cmake with colcon packages from their master branches.

[sloretz]

For debugging, it may be helpful to add print(os.environ['PYTHONPATH']) and print(sys.path) here:

https://github.com/ament/ament_cmake/blob/2bf27ce5ad01af340e1f4efd44232c1067d0dbda/ament_cmake_core/cmake/package_templates/templates_2_cmake.py#L19

It would also be helpful to print the content of the directories in sys.path. Comparing that info with the working colcon version and the non-working one would give some clues as to what's going on.

[mjcarroll]

Broken configuration:

sys.path=['/root/ros2_rolling/src/ament/ament_cmake/ament_cmake_core/cmake/package_templates', '/root/ros2_rolling/install/ament_package/local/lib/python3.10/dist-packages', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages']
os.environ=/root/ros2_rolling/install/ament_package/local/lib/python3.10/dist-packages

$ tree /root/ros2_rolling/src/ament/ament_cmake/ament_cmake_core/cmake/package_templates
/root/ros2_rolling/src/ament/ament_cmake/ament_cmake_core/cmake/package_templates
`-- templates_2_cmake.py

$ tree /root/ros2_rolling/install/ament_package/local/lib/python3.10/dist-packages
/root/ros2_rolling/install/ament_package/local/lib/python3.10/dist-packages

0 directories, 0 files

[mjcarroll]

Working configuration:

sys.path=['/root/ros2_rolling/src/ament/ament_cmake/ament_cmake_core/cmake/package_templates', '/root/ros2_rolling/install/ament_package/lib/python3.10/site-packages', '/usr/lib/python310.zip', '/usr/lib/python3.10', '/usr/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/dist-packages', '/usr/lib/python3/dist-packages']
os.environ=/root/ros2_rolling/install/ament_package/lib/python3.10/site-packages

$ tree /root/ros2_rolling/src/ament/ament_cmake/ament_cmake_core/cmake/package_templates
/root/ros2_rolling/src/ament/ament_cmake/ament_cmake_core/cmake/package_templates
`-- templates_2_cmake.py

$ tree  /root/ros2_rolling/install/ament_package/lib/python3.10/site-packages
/root/ros2_rolling/install/ament_package/lib/python3.10/site-packages
|-- ament_package
|   |-- __init__.py
|   |-- __pycache__
|   |   |-- __init__.cpython-310.pyc
|   |   `-- templates.cpython-310.pyc
|   |-- template
|   |   |-- __init__.py
|   |   |-- __pycache__
|   |   |   `-- __init__.cpython-310.pyc
|   |   |-- environment_hook
|   |   |   |-- __init__.py
|   |   |   |-- __pycache__
|   |   |   |   `-- __init__.cpython-310.pyc
|   |   |   |-- ament_prefix_path.bat
|   |   |   |-- ament_prefix_path.sh
|   |   |   |-- library_path.sh
|   |   |   |-- path.bat
|   |   |   |-- path.sh
|   |   |   |-- pkg_config_path.bat
|   |   |   |-- pkg_config_path.sh
|   |   |   |-- pythonpath.bat.in
|   |   |   `-- pythonpath.sh.in
|   |   |-- isolated_prefix_level
|   |   |   |-- __init__.py
|   |   |   |-- __pycache__
|   |   |   |   |-- __init__.cpython-310.pyc
|   |   |   |   `-- _order_isolated_packages.cpython-310.pyc
|   |   |   |-- _order_isolated_packages.py
|   |   |   |-- local_setup.bash
|   |   |   |-- local_setup.bat.in
|   |   |   |-- local_setup.sh.in
|   |   |   `-- local_setup.zsh
|   |   |-- package_level
|   |   |   |-- __init__.py
|   |   |   |-- __pycache__
|   |   |   |   `-- __init__.cpython-310.pyc
|   |   |   |-- local_setup.bash.in
|   |   |   |-- local_setup.bat.in
|   |   |   |-- local_setup.sh.in
|   |   |   `-- local_setup.zsh.in
|   |   `-- prefix_level
|   |       |-- __init__.py
|   |       |-- __pycache__
|   |       |   |-- __init__.cpython-310.pyc
|   |       |   `-- _local_setup_util.cpython-310.pyc
|   |       |-- _local_setup_util.py
|   |       |-- local_setup.bash
|   |       |-- local_setup.bat.in
|   |       |-- local_setup.sh.in
|   |       |-- local_setup.zsh
|   |       |-- setup.bash
|   |       |-- setup.bat.in
|   |       |-- setup.sh.in
|   |       `-- setup.zsh
|   `-- templates.py
`-- ament_package-0.14.0-py3.10.egg-info
    |-- PKG-INFO
    |-- SOURCES.txt
    |-- dependency_links.txt
    |-- top_level.txt
    `-- zip-safe

0 directories, 0 files

[sloretz]

Hmm, where did that local come from?

/root/ros2_rolling/install/ament_package/local/lib/python3.10/dist-packages
/root/ros2_rolling/install/ament_package/lib/python3.10/site-packages

[sloretz]

local is sneaking in in the build of ament_package from the shell scripts generated by Colcon

# No comment in this file
install/ament_package/share/ament_package/hook/pythonpath.dsv:prepend-non-duplicate;PYTHONPATH;local/lib/python3.10/dist-packages
# generated from colcon_powershell/shell/template/hook_prepend_value.ps1.em
install/ament_package/share/ament_package/hook/pythonpath.ps1:colcon_prepend_unique_value PYTHONPATH "$env:COLCON_CURRENT_PREFIX\local/lib/python3.10/dist-packages"
# generated from colcon_core/shell/template/hook_prepend_value.sh.em
install/ament_package/share/ament_package/hook/pythonpath.sh:_colcon_prepend_unique_value PYTHONPATH "$COLCON_CURRENT_PREFIX/local/lib/python3.10/dist-packages"

[sloretz]

Path containing local must be from the variable subdirectory https://github.com/colcon/colcon-core/blob/37cf8aaee99b9a65c8dc5579abc7d43c631a1f97/colcon_core/shell/template/hook_prepend_value.sh.em#L10

subdirectory is an argument to the member function create_hook_prepend_value https://github.com/colcon/colcon-core/blob/37cf8aaee99b9a65c8dc5579abc7d43c631a1f97/colcon_core/shell/sh.py#L114-L116

That function appears to be called here when shell extensions are asked to create environment hooks https://github.com/colcon/colcon-core/blob/37cf8aaee99b9a65c8dc5579abc7d43c631a1f97/colcon_core/shell/__init__.py#L417-L419

subdirectory is passed in as an argument to create_environment_hook https://github.com/colcon/colcon-core/blob/37cf8aaee99b9a65c8dc5579abc7d43c631a1f97/colcon_core/shell/__init__.py#L368-L370

I think the call we care about is the one creating the PYTHONPATH environment hook https://github.com/colcon/colcon-core/blob/37cf8aaee99b9a65c8dc5579abc7d43c631a1f97/colcon_core/environment/pythonpath.py#L29-L31

Which gets us to this logic for creating the subdirectory variable

https://github.com/colcon/colcon-core/blob/37cf8aaee99b9a65c8dc5579abc7d43c631a1f97/colcon_core/environment/pythonpath.py#L21-L28

This was recently changed in #483 (@cottsay FYI) because distutils is deprecated.

I don't know the exact arguments passed to sysconfig, but I strongly suspect this is where local comes from

>>> import sysconfig
>>> sysconfig.get_path('purelib', vars={'base': '.'})
'local/lib/python3.10/dist-packages'
>>> from distutils.sysconfig import get_python_lib
<stdin>:1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
<stdin>:1: DeprecationWarning: The distutils.sysconfig module is deprecated, use sysconfig instead
>>> get_python_lib(prefix='.')
'./lib/python3.10/site-packages'

[sloretz]

I suspect it can be fixed by using the posix_prefix scheme, though I'm not 100% sure what that means from reading the docs. It just gives the answer I think we want.

>>> sysconfig.get_path('purelib', vars={'base': '.'}, scheme='posix_prefix')
'lib/python3.10/site-packages'
>>> sysconfig.get_path('purelib', vars={'base': '.'}, scheme='posix_home')
'lib/python'
>>> sysconfig.get_path('purelib', vars={'base': '.'}, scheme='posix_user')
'/root/.local/lib/python3.10/site-packages'

[sloretz]

The default scheme is posix_local, which isn't documented

>>> sysconfig.get_default_scheme()
'posix_local'

[pablogs9]

Just for confirmation, this buggy version of colcon-core is being released along with the last Rolling version:

root@571bda4a315a:/# apt show python3-colcon-core
Package: python3-colcon-core
Version: 0.8.1-1
Priority: optional
Section: python
Maintainer: Dirk Thomas <web@dirk-thomas.net>
Installed-Size: 421 kB
Depends: python3-distlib, python3-empy, python3-pkg-resources, python3-pytest, python3-pytest-cov, python3:any (>= 3.6~), python3-setuptools
Suggests: python3-pytest-repeat, python3-pytest-rerunfailures
Download-Size: 62.2 kB
APT-Manual-Installed: no
APT-Sources: http://packages.ros.org/ros2/ubuntu jammy/main amd64 Packages
Description: Command line tool to build sets of software packages.
 colcon - collective construction
 ================================
 .
 ``colcon`` is a command line tool to improve the workflow of building, testing and using multiple software packages.
 It automates the process, handles the ordering and sets up the environment to use the packages.
 .
 For more information see `colcon.readthedocs.io <https://colcon.readthedocs.io>`_.

Do you have an estimated release date and sync with rolling packages? This bug is breaking all the micro-ROS build system in Rolling.

[nuclearsandwich]

Do you have an estimated release date and sync with rolling packages? This bug is breaking all the micro-ROS build system in Rolling.

colcon is part of the "bootstrap" layer of ROS and ROS 2 packages which are imported directly from the ros_bootstrap repository. (This repository is only intended to be consumed directly by build farm instances and should not be configured on any ROS developer workstation or production deployment. All packages from the bootstrap repository are imported when made available and shortly after deployed to the testing and main ROS and ROS 2 repositories regardless of sync status. So a Rolling sync is not required for the colcon package release, which has been done and imported into the ROS repositories with these linked import jobs.