Enable SDK for ROS2 Rolling

[sgstreet]

This PR tries to resurrect PR #988 for Rolling to enable OpenEmbedded SDK cross-compilation. These changes allow demos, examples and examples_interfaces packages to be built using the resulting SDK. To build the SDK use:

$ bitbake ros2-image-sdktest -c do_populate_sdk

Install the resulting SDK, setup a ROS2 workspace containing demos, examples and example_interfaces and build the workspace using the follow script:

export ROS_VERSION=2
export ROS_PYTHON_VERSION=3
export ROS_AUTOMATIC_DISCOVERY_RANGE=SUBNET
export ROS_DISTRO=rolling
export PYTHONPATH=$OECORE_NATIVE_SYSROOT/usr/lib/python3.12/site-packages:$OECORE_TARGET_SYSROOT/opt/ros/rolling/lib/python3.12/site-packages
export AMENT_PREFIX_PATH=$OECORE_TARGET_SYSROOT/opt/ros/rolling
usr/share:$OECORE_TARGET_SYSROOT/opt/ros/rolling/share:$OECORE_TARGET_SYSROOT/opt/ros/rolling/lib/cmake"

colcon build \
--merge-install \
--install-base $PWD/install/arm64 \
--cmake-args \
" -DCMAKE_TOOLCHAIN_FILE=$CMAKE_TOOLCHAIN_FILE" \
" -DCMAKE_STAGING_PREFIX=$PWD/install/arm64" \
" -DBUILD_TESTING=OFF"

Install the $PWD/install/arm64 on the target device. Setup the device workspace:

$ . /opt/ros/rolling/setup.bash
$ . ./arm64/local_setup.bash

Run any of the rclcpp examples to demonstrate the valid cross-compilation.

[sgstreet]

The python nodes are not functional:

root@raspberrypi5:~# ros2 run demo_nodes_py talker
Traceback (most recent call last):
  File "/opt/ros/rolling/bin/ros2", line 33, in <module>
    sys.exit(load_entry_point('ros2cli==0.33.0', 'console_scripts', 'ros2')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ros/rolling/lib/python3.12/site-packages/ros2cli/cli.py", line 91, in main
    rc = extension.main(parser=parser, args=args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ros/rolling/lib/python3.12/site-packages/ros2run/command/run.py", line 70, in main
    return run_executable(path=path, argv=args.argv, prefix=prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ros/rolling/lib/python3.12/site-packages/ros2run/api/__init__.py", line 64, in run_executable
    process = subprocess.Popen(cmd)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.12/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/root/arm64/lib/demo_nodes_py/talker'

The file /home/root/arm64/lib/demo_nodes_py/talker exists and contains the expected python code. Does someone have an idea on how to debug this?

[jiaxshi]

Hi @sgstreet Greate job! I have done similar work on humble, I suggest to add TOOLCHAIN_HOST_TASK in a include file

+require conf/distro/include/ros2-sdk.inc

Could you please full build https://github.com/ros2/examples or https://github.com/ros2/demos with SDK env?

[sgstreet]

Could you please full build https://github.com/ros2/examples or https://github.com/ros2/demos with SDK env?

Yes. I can build demos, example_interfaces and example ros2 packages using the SDK env. The C++ nodes are functional,unfortunately the python packages are not, see above.

[jiaxshi]

The python nodes are not functional:

root@raspberrypi5:~# ros2 run demo_nodes_py talker
Traceback (most recent call last):
  File "/opt/ros/rolling/bin/ros2", line 33, in <module>
    sys.exit(load_entry_point('ros2cli==0.33.0', 'console_scripts', 'ros2')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ros/rolling/lib/python3.12/site-packages/ros2cli/cli.py", line 91, in main
    rc = extension.main(parser=parser, args=args)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ros/rolling/lib/python3.12/site-packages/ros2run/command/run.py", line 70, in main
    return run_executable(path=path, argv=args.argv, prefix=prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/ros/rolling/lib/python3.12/site-packages/ros2run/api/__init__.py", line 64, in run_executable
    process = subprocess.Popen(cmd)
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.12/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/root/arm64/lib/demo_nodes_py/talker'

The file /home/root/arm64/lib/demo_nodes_py/talker exists and contains the expected python code. Does someone have an idea on how to debug this?

Hi @sgstreet
You can check the python shebang in python executor. HOST path is included like:

#!/local/mnt2/workspace/install_dir/sysroots/x86_64-qcomsdk-linux/usr/bin/python3

[sgstreet]

@jiaxshi Still hacking on the shebang path.

[jiaxshi]

@jiaxshi Still hacking on the shebang path.

python executor can be found after changing shebang path. And one more thing, if you build python-library, you should add PYTHON_SOABI to rename library which is named with "x86".

-DPYTHON_SOABI=cpython-310-aarch64-linux-gnu

You can refer to #1204 for details.

[sgstreet]

@jiaxshi here is the contents of my talker executor (are these correct words?)

#!/home/stephen/Workspace/ros2-dev/sdk/sysroots/x86_64-oesdk-linux/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'demo-nodes-py==0.34.2','console_scripts','talker'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'demo-nodes-py==0.34.2'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point

def importlib_load_entry_point(spec, group, name):
    dist_name, _, _ = spec.partition('==')
    matches = (
        entry_point
        for entry_point in distribution(dist_name).entry_points
        if entry_point.group == group and entry_point.name == name
    )
    return next(matches).load()

globals().setdefault('load_entry_point', importlib_load_entry_point)

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(load_entry_point('demo-nodes-py==0.34.2', 'console_scripts', 'talker')())

As you pointed out the shebang is clearly wrong. Can you point out the code which generates this talker file? Maybe I can figure out which path is wrong.

[sgstreet]

python executor can be found after changing shebang path. And one more thing, if you build python-library, you should add PYTHON_SOABI to rename library which is named with "x86".

-DPYTHON_SOABI=cpython-310-aarch64-linux-gnu

You can refer to #1204 for details.

I tried this, how should I verify the impact?

Further, for none python ROS2 packages the following warnings are generated:

CMake Warning:
  Manually-specified variables were not used by the project:

    PYTHON_SOABI

which is harmless but annoying. I think we need another place to add the PYTHON_SOABI.

[jiaxshi]

I think the shebang path is added by native python interpreter.

CMake Warning:
  Manually-specified variables were not used by the project:

    PYTHON_SOABI

The warning is because is not used by none-python packages. If there's way to add the default value to native interpreter is better.

[sgstreet]

I think the shebang path is added by native python interpreter. Are you saying the native python3 is adding the shebang? I would have thought either the python setup or some ament_python make script was doing this but I'm not sure where to look. Maybe I'm confused? I have limit python skills.

CMake Warning:

  Manually-specified variables were not used by the project:

    PYTHON_SOABI

The warning is because is not used by none-python packages. If there's way to add the default value to native interpreter is better.

Or maybe superflores could add during recipe generation.

[sgstreet]

@jiaxshi here is the contents of my talker executor (are these correct words?)

#!/home/stephen/Workspace/ros2-dev/sdk/sysroots/x86_64-oesdk-linux/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'demo-nodes-py==0.34.2','console_scripts','talker'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'demo-nodes-py==0.34.2'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point

def importlib_load_entry_point(spec, group, name):
    dist_name, _, _ = spec.partition('==')
    matches = (
        entry_point
        for entry_point in distribution(dist_name).entry_points
        if entry_point.group == group and entry_point.name == name
    )
    return next(matches).load()

globals().setdefault('load_entry_point', importlib_load_entry_point)

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(load_entry_point('demo-nodes-py==0.34.2', 'console_scripts', 'talker')())

As you pointed out the shebang is clearly wrong. Can you point out the code which generates this talker file? Maybe I can figure out which path is wrong.

I patched colcon build to generate a better/correct shebang for python packages builds. See https://github.com/ros/meta-ros/pull/1203/commits/5965ca671826a80127627de67f951a4cb3264003

[jiaxshi]

@jiaxshi here is the contents of my talker executor (are these correct words?)

#!/home/stephen/Workspace/ros2-dev/sdk/sysroots/x86_64-oesdk-linux/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'demo-nodes-py==0.34.2','console_scripts','talker'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'demo-nodes-py==0.34.2'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point

def importlib_load_entry_point(spec, group, name):
    dist_name, _, _ = spec.partition('==')
    matches = (
        entry_point
        for entry_point in distribution(dist_name).entry_points
        if entry_point.group == group and entry_point.name == name
    )
    return next(matches).load()

globals().setdefault('load_entry_point', importlib_load_entry_point)

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(load_entry_point('demo-nodes-py==0.34.2', 'console_scripts', 'talker')())

As you pointed out the shebang is clearly wrong. Can you point out the code which generates this talker file? Maybe I can figure out which path is wrong.

I patched colcon build to generate a better/correct shebang for python packages builds. See 5965ca6

Hi @sgstreet
Is the code based on your 5965ca6 commit? If so, it's not correct. As you see,

#!/home/stephen/Workspace/ros2-dev/sdk/sysroots/x86_64-oesdk-linux/usr/bin/python3

Talker executor is running on target(e.g. RPI-4 board) device, the path python3 executor is /usr/bin/python3 not /home/stephen/Workspace/ros2-dev/sdk/sysroots/x86_64-oesdk-linux/usr/bin/python3 which is your HOST python3 executor of SDK. The right output is

#!/usr/bin/python3

without HOST path prefix(/home/stephen/Workspace/ros2-dev/sdk/sysroots/x86_64-oesdk-linux) included.

[sgstreet]

@jiaxshi Sorry for the confusing and incomplete post. Building python nodes with 5965ca results in the following talker code:

#!/usr/bin/env python3
# EASY-INSTALL-ENTRY-SCRIPT: 'demo-nodes-py==0.34.2','console_scripts','talker'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'demo-nodes-py==0.34.2'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point

def importlib_load_entry_point(spec, group, name):
    dist_name, _, _ = spec.partition('==')
    matches = (
        entry_point
        for entry_point in distribution(dist_name).entry_points
        if entry_point.group == group and entry_point.name == name
    )
    return next(matches).load()

globals().setdefault('load_entry_point', importlib_load_entry_point)

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(load_entry_point('demo-nodes-py==0.34.2', 'console_scripts', 'talker')())

Which has the correct SHEBANG line.

[sgstreet]

Joy, cpp and python code rolling demos@0.34.1 and examples@0.20.1 are functional against origin/scarthgap-next using oe scarthgap release on a raspberry pi 5.

[sgstreet]

@robwoolley @jiaxshi This is almost ready to go, what are the next steps for getting this pull request merged? Jazzy, Humble? What about Kirkstone, master, Scarthgap? What addition changes are needed?

[jiaxshi]

Joy, cpp and python code rolling demos@0.34.1 and examples@0.20.1 are functional against origin/scarthgap-next using oe scarthgap release on a raspberry pi 5.

That's cool.

To merge the PR, my opinion is that we should test on other ROS2 distros (Humble, Jazzy, Iron, Rolling) and then propagate to the other branch (Kirkstone).

And one more topic is image recipe: meta-ros-common/recipes-core/images/ros2-image-sdktest.bb . How to generate SDK based on customized image recipe with platform dependency included? For example, if i want to build a board/platform(raspberry Pi, RB3 gen2, Nvidia) specific image recipe and generate SDK with both ROS and platform support(e.g. GPU SDK) included. @robwoolley Do you have any thoughts on this topic? If the scope of SDK is platform independent, we can ignore this.

[sgstreet]

To merge the PR, my opinion is that we should test on other ROS2 distros (Humble, Jazzy, Iron, Rolling) and then propagate to the other branch (Kirkstone).

I will work on this over the next couple of days, I suspect that Humble and Jazzy will need some specific patches for fastrtp and tinyxml. I was planning to ignore Iron as it will EOL in November. @robwoolley Thoughts?

[robwoolley]

Merged into master-next