More complete Cross-compile instructions

jpsamper2009 commented 6 years ago

I've been meaning to create this post for a while since

I have found a way to cross-compile tests, python packages, and with security ON
But, it seems awfully hacky and I want to get some feedback on how to make it less so

I have added "Notes:" for the steps where I'm definitely doing something questionable.

For completeness, I have copied the existing documentation and made the necessary changes:

Throughout the documentation I assume you have created a directory ros2_ws in home (i.e ~/ros2_ws)

Setup the development environment

The normal setup described in the following link is required for the cross-compilation -> https://github.com/ros2/ros2/wiki/Linux-Development-Setup

Get the source

Create a workspace and clone all repos:

mkdir -p ros2_ws/src
cd ros2_ws
wget https://raw.githubusercontent.com/ros2/ros2/release-latest/ros2.repos
wget https://raw.githubusercontent.com/ros2-for-arm/ros2/master/ros2-for-arm.repos
wget https://raw.githubusercontent.com/ros2-for-arm/ros2/master/aarch64_toolchainfile.cmake
vcs-import src < ros2.repos
vcs-import src < ros2-for-arm.repos

Get an aarch64 toolchain and export it

sudo apt install g++-aarch64-linux-gnu gcc-aarch64-linux-gnu

# If you are not using a custom toolchain
export PATH="<path to toolchain>/bin:$PATH"
export CROSS_COMPILE=aarch64-linux-gnu-

Ignore packages that won't build with Python/Security

The following command disables the build of such packages by adding an empty file called COLCON_IGNORE in their base directory:

touch \
  src/ros/resource_retriever/COLCON_IGNORE \
  src/ros-perception/laser_geometry/COLCON_IGNORE \
  src/ros2/demos/image_tools/COLCON_IGNORE \
  src/ros2/demos/intra_process_demo/COLCON_IGNORE \
  src/ros2/demos/pendulum_control/COLCON_IGNORE \
  src/ros2/demos/pendulum_msgs/COLCON_IGNORE \
  src/ros2/geometry2/COLCON_IGNORE \
  src/ros2/kdl_parser/COLCON_IGNORE \
  src/ros2/orocos_kinematics_dynamics/COLCON_IGNORE \
  src/ros/pluginlib/pluginlib/COLCON_IGNORE \
  src/ros2/rmw_connext/COLCON_IGNORE \
  src/ros2/rmw_opensplice/COLCON_IGNORE \
  src/ros2/robot_state_publisher/COLCON_IGNORE \
  src/ros2/ros1_bridge/COLCON_IGNORE \
  src/ros2/rosidl_typesupport_connext/COLCON_IGNORE \
  src/ros2/rosidl_typesupport_opensplice/COLCON_IGNORE \
  src/ros2/realtime_support/rttest/COLCON_IGNORE \
  src/ros2/rviz/COLCON_IGNORE \
  src/ros2/urdf/COLCON_IGNORE \
  src/ros2/urdfdom/COLCON_IGNORE \
  src/ros2/system_tests/COLCON_IGNORE

Create a target filesystem

Note: This is the first hacky step. Maybe someone has a better idea on how to create this target filesystem.

On an aarch64 machine:

Create a Dockerfile to install all dependencies needed to build ros2
docker build the image
Start a container using the image
Use docker export to get a tar-ball with an aarch64 filesystem
Untar this file into a folder names rootfs, delete all non-Python, non-OpenSSL files
- Note: This may not be necessary. I originally did this because CMAKE_SYSROOT was leading CMake to look for executables like make in rootfs which were not executable since they were built aarch64 executables
Tar rootfs and copy it to an x86_64 machine

Place this file next to ros2_ws (ie. ~/rootfs/)

Trigger a build

$ colcon build \
           --merge-install \
           --cmake-force-configure \
           --cmake-args \
               --no-warn-unused-cli \
               -DCMAKE_BUILD_TYPE=Debug \
               -DCMAKE_TOOLCHAIN_FILE="$(pwd)/aarch64_toolchainfile.cmake" \
               -DTHIRDPARTY=ON \
               -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON \
               -DPYTHON_INCLUDE_DIR="$(pwd)/../rootfs/usr/include/;$(pwd)/../rootfs/usr/include/python3.5m/" \
               -DPYTHON_LIBRARY="$(pwd)/../rootfs/usr/lib/aarch64-linux-gnu/libpython3.5m.so" \
               -DSECURITY=ON \
               -DOPENSSL_INCLUDE_DIR="$(pwd)/../rootfs/usr/include;$(pwd)/../rootfs/usr/include/aarch64-linux-gnu" \
               -DOPENSSL_SSL_LIBRARY="$(pwd)/../rootfs/usr/lib/aarch64-linux-gnu/libssl.so" \
               -DOPENSSL_CRYPTO_LIBRARY="$(pwd)/../rootfs/usr/lib/aarch64-linux-gnu/libcrypto.so"

Note: I tried using CMAKE_SYSROOT instead of specifying all these paths, but didn't have much luck.
Note: This only works with --merge-install
- I couldn't get -DCMAKE_BUILD_RPATH=... to work

Cleanup some references to rootfs (i.e. The Hackiest Step)

$ for f in $(find ./install ./build -name "*cpython*.so"); do \
         ln -s $(basename $f) ${f/.cpython*/.so}; \
     done
$ ROOTFS_PATH="$(realpath ../rootfs)"
$ sed -i "s+$ROOTFS_PATH++g" "$(find ./install -name "fastrtpsTargets.cmake")"

These steps are needed to make sure that other packages will find the correct libraries when ROS 2 is deployed:
- The first workaround (for f in ...) is required because cython generates some dynamic libraries and names them based on the architecture of the current machine and not the target architecture
  - For example, it generates _rclpy.cpython-x86_64-linux-gnu.so, but if you run file _rclpy.cpython-x86_64-linux-gnu.so, it says ELF 64-bit LSB shared object, ARM aarch64
- The second workaround (sed -i ...) strips the path to rootfs such that the paths to OpenSSL libraries match the paths on the aarch64 machine

Running Tests

To run the tests, create an identical path on your target aarch64 machine (e.g. "/home/<your-username>/ros2_ws/) and copy the src, build, and install folders.

cd ~
ln -s / rootfs  # !!!Note:!!! Obvious hack - but I think it's only needed for stuff in colcon_command_prefix scripts
cd ~/ros2_ws
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$(pwd)/install/lib"
colcon test --merge-install

Installation

On the target machine, copy the contents of install folder to, for example, /opt/ros2/bouncy/
Run /opt/ros2/bouncy/setup.bash to use it
It should be possible to use ros2 run to start nodes
It should be possible to natively compile a simple application

pokitoz commented 6 years ago

Thanks, it looks great and I will have a deeper look.

It is great that you also gave the steps to enable not only python but also OpenSSL to possibly enable secure ROS2. I also have a hacky solution which uses Qemu instead of Docker (but the result should be similar), however your steps looks more clear. I will try to merge both solution and remove (if possible) the hacky parts.

I think that creating the file-system can be considered as a normal step as we need lots of dependencies for python/openssl support.

About the "Cleanup some references to rootfs (i.e. The Hackiest Step)", I was looking at https://stackoverflow.com/questions/38523941/change-cythons-naming-rules-for-so-files but your solution looks simpler.

dejanpan commented 5 years ago

@filiperinaldi since you just mentioned this topic in the TSC meeting today. Will any of the issues described above be tackled for Crystal?

filiperinaldi commented 5 years ago

Hi @dejanpan, yes, that is the goal. Here is the thread @lmayencourt started on Discourse: https://discourse.ros.org/t/ros2-cross-compilation/6834

ros2-for-arm / ros2