Refactor cross-compilation and add Raspbian support

alsora commented 5 years ago

This PR completely refactors the cross_compile repository, using the format of its counter-part developed by iRobot. Moreover it adds support for Raspbian.

The Raspbian cross-compilation has been extensively tested in crystal, dashing and the current ROS2 master. Moreover, this way in which the framework is organized allows to add support for new architectures without having to duplicate code.

How it works

The cross-compilation happens inside a Ubuntu 18.04 Docker environment. This allows to avoid having to install dependencies on your machine. This Docker environment is mainly defined in the docker_environments/Dockerfile_base file. This Dockerfile provides general dependencies and ROS2 specific ones. It does not provide any particular toolchain or compiler. For each target architecture, there is an additional Dockerfile that includes the base one. For example the docker_environments/Dockerfile_raspbian file. This will be the one used for cross-compiling for Raspbian. In addition to the base dockerfile, it provides the required compiler and any additional configuration (specific to Raspbian) that may be needed.
The toolchains directory contains the generic_linux.cmake file, i.e. a toolchain cmake file where all the entries are provided through environment variables. Additionally in this directory you find bash scripts for each supported architecture that are used to set the required environment variables.
The sysroots directory contains files for each supported architecture that can be used to generate a sysroot with the required ROS2 dependencies. Both the 2 supported architectures use a Dockerfile to generate the sysroot.
The ignore_pkgs directory contains a bash scripts for each ros2 distribution that ignore some packages that may fail or that are useless for a cross-compiled target. This directory can be expanded with architecture-specific ignore files.

How to add a new architecture

Mandatory

Create a docker_environments/Dockerfile_my-architecture where you install the compiler.
Create a toolchains/my-architecture.sh that exports the required cross-compilation flags.

Optional

Create a script for generating the sysroot.

Notes

The cross-compilation happens in a Dockerfile, but this is a standard x86-64 dockerfile so there is no degradation of the performances. The cross-compilation works also for single packages, without having to add them to a complete ROS2 SDK, as long as the already cross-compiled SDK is copied to the sysroot.

@filiperinaldi @christianrauch

christianrauch commented 5 years ago

I successfully built most of the packages with a custom Rasbian buster rootfs that I created using: sudo qemu-debootstrap --no-check-gpg --arch armhf buster raspbian_buster_rootfs http://archive.raspbian.org/raspbian.

I encountered an issue with osrf_testing_tools_cpp:

make[2]: *** No rule to make target '/root/sysroot/usr/lib/arm-linux-gnueabihf/libdl.so', needed by 'src/memory_tools/libmemory_tools.so'.  Stop.
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [src/memory_tools/CMakeFiles/memory_tools.dir/all] Error 2
make: *** [all] Error 2
---
Failed   <<< osrf_testing_tools_cpp [ Exited with code 2 ]

and needed to remove the package from the workspace.

christianrauch commented 5 years ago

I am getting Illegal instruction when running any of the compiled executables on RPi 1 hardware, e.g. lib/examples_rclcpp_minimal_publisher/publisher_member_function. ros2 topic list gives many errors of the form Failed to load entry point '[...]': No module named 'rclpy._rclpy'.

The command file reports ELF 32-bit LSB shared object, ARM, EABI5 version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0 which seems to be right. Is a specific toolchain file for the Raspberry Pi 1 (A, B, Zero) required? E.g. one of the toolchains listed at https://github.com/raspberrypi/tools/tree/master/arm-bcm2708.

alsora commented 5 years ago

I only tested with Raspbian stretch on RaspberryPi 2 and RaspberryPi 3. For these platforms the toolchains that you linked were not working.

I'll try to have a look at this, but I don't have the hardware to reproduce. I can check buster on RaspberryPi3

christianrauch commented 5 years ago

I tested with the automatic_cross_compile.sh on RPi1 and I get the same Illegal instruction error. Maybe the Raspberry Pi 1 CPU needs another toolchain.

christianrauch commented 5 years ago

The same Raspbian buster image and the two ROS2 builds (automatic and custom) work perfectly on a Raspberry Pi2. I assume that the standard toolchains in ubuntu (g++-6-arm-linux-gnueabihf) only build executables that are compatible with armv7 and armv8 in the Pi 2 and 3, but not the armv6 in the Pi 1. I was able to integrate the raspberrypi/tools toolchain but the older gcc 4.8 fails to build FastRTPS because it does not support std::put_time. Since it doesn't seem there will be updates to this toolchain, I was looking into building with crosstool-ng using the docker image at https://hub.docker.com/r/idein/cross-rpi but encountered other building issues.

It might be useful to mention that the standard toolchain will not be able to build executables for the Pi 1.

alsora commented 5 years ago

I added a note that this does not work with ARMv6

I was reading that this toolchain should do the job for RaspberryPi 1 https://github.com/raspberrypi/tools/tree/master/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf

I will make some tests and in case I can differentiate the raspbian target according to which toolchain it needs.

christianrauch commented 5 years ago

Yes, the toolchains in this repo should be able to compile compatible binaries, but it's "very" old and doesn't fully support C++11. I was able to compile partially using the gcc-linaro-arm-linux-gnueabihf-raspbian-x64 toolchain, but it fails for FastRTPS because std::put_time is only available from GCC 5 on: https://stackoverflow.com/a/14137287.

pokitoz commented 5 years ago

Have you had the chance to look at https://github.com/ros2/cross_compile/pull/20 and the associated design document? It looks to aim at the same goal: refactor this repo to ease cross-compilation but with a different approach

emersonknapp commented 5 years ago

Closing this PR as out of date. Checking on raspbian support in the new implementation would be very welcome.

ros-tooling / cross_compile