Cross-compilation (host x86_64 linux - target aarch64 linux)

[photex]

Howdy!

It appears that the associated sys crate for gstreamer relies on pkg-config. This seems to mean it'd be almost impossible to cross-compile using what's available via apt (since the dev packages cannot co-exist).

I'm developing a system that has components running on both x86_64 systems and embedded aarch64 systems (nvidia jetson) and I'd love to use this without having to kludge something together.

Have you tried to build any of these crates using an alternate rust target? Any advice?

Cheers! Chip

[sdroege]

This should in theory be fixed just now with https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=823307

pkg-config itself is cross-compile aware (see also e.g. the sysroot environment variable, the host-triplet variants of pkg-config and the .pc files being installed into an arch-specific directory) and you're going to have a worse time with other options than pkg-config. Apart from that, finding the GStreamer libraries correctly is supported by the GStreamer project only via the pkg-config files, no other mechanism.

Is there any other problem you're running into after installing the aarch64 version of the -dev packages?

[photex]

Ok, I feel silly for not looking at the build.rs first but now I see I can specify GTK_LIB_DIR. Problem isn't solved however and I'd love to hear any thoughts:

After having installed the aarch64 gcc and gstreamer libs for arm64 and running: GTK_LIB_DIR=/usr/lib/aarch64-linux-gnu cargo build --target=aarch64-unknown-linux-gnu

I end up with the following error:

= note: /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: cannot find -lgstreamer-1.0
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: cannot find -lgobject-2.0
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: cannot find -lglib-2.0
          /usr/lib/gcc-cross/aarch64-linux-gnu/5/../../../../aarch64-linux-gnu/bin/ld: cannot find -lgobject-2.0
          collect2: error: ld returned 1 exit status

I see that -L /usr/lib/aarch64-linux-gnu is being correctly passed to rustc (very many times even). The problem appears to be that pretty much every lib except libglib-2.0 is present. Apt reports this lib to be installed.

[amotl]

Hi there,

while we don't feel qualified to add something reasonable to this discussion (still learning), we also wrestled with cross compiling gstreamer-rs against MIPS the other day, which might also apply to aarch64.

While doing that, we collected some information about the troubleshooting process [1], which you might find helpful.

We can barely remember we also had troubles making pkg-config find glib and in fact there are some traces about that in the document [1]. We found these environment variables to be helpful:

export TARGET=mips-unknown-linux-gnu
export RUST_BACKTRACE=1
export PKG_CONFIG_ALLOW_CROSS=1
export PKG_CONFIG_PATH=/usr/lib/mips-linux-gnu/pkgconfig

Especially, cargo build --target=mips-unknown-linux-gnu seemed to be not sufficient to propagate the designated target down the compiling and linking chain, so we added an additional export TARGET=mips-unknown-linux-gnu to satisfy all things involved.

At some time, we could make pkg-config find the appropriate references to glib:

"pkg-config" "--libs" "--cflags" "glib-2.0 >= 2.32"
-I/usr/include/glib-2.0 -I/usr/lib/mips-linux-gnu/glib-2.0/include -L/usr/lib/mips-linux-gnu -lglib-2.0

Sorry when talking about obvious things here but we dared to share our outcome anyway, at least others might find this helpful as there have been some speed bumps for us who don't do system programming and cross compilation on a daily basis.

Keep up the spirit!

Cheers, Andreas.

[1] https://github.com/hiveeyes/saraswati/blob/master/doc/rust-cross-troubleshooting.rst

---

P.S.: This should have been "first things first": Kudos to @sdroege and the whole GStreamer and gstreamer-rs communities for conceiving these fine pieces of software. Thanks for your work on this!

[sdroege]

GTK_LIB_DIR is a hack that should ideally go away, see https://github.com/gtk-rs/gir/pull/631

The real solution is like what @amotl said. You have to make sure to set the correct environment variables so that pkg-config uses the .pc files for your target platform and prints the correct paths as expected by the linker.

@amotl @photex If you want to prepare a PR that updates the README.md accordingly with the relevant steps to make this work, please feel free to do so. This should be documented as it's apparently not obvious :)

[photex]

The only bummer about that bug-report is that the solution will not be applicable in practice for the foreseeable future since it wouldn't be something I can install via apt for both architectures on the platforms used for this project. The GTK_LIB_DIR hack at least offered some hope. ;) The other problem there is that I believe we're tied to gstreamer-1.8 on the Jetson since we definitely require the various nvidia plugins (Maybe the sources for these are available somewhere though).

I'm willing to take one for the team and work out how to build the frankly enormous number of libraries for each arch if there was an easy way to automate that. We currently use CMake as our build system (with some simple macros to handle/integrate cargo). I'm willing to explore Meson if it makes life much easier.

Unless I'm mistaken and there is actually something like the Qt installer with all the various gnome stuff built for some spread of architectures. :D

Anyway, yes I'd be definitely willing to submit a PR to the README.md if I can come up with a solution that isn't specific to my project. And just to echo @amotl, it's an amazing amount of really solid work done to put all these crates together and the effort is very much appreciated!

[photex]

Short addendum: The reason that the arm64 glib library doesn't get installed:

kopernikus@bigboy examples (master*) $ sudo dpkg -i libglib2.0-0_2.48.2-0ubuntu4_arm64.deb 
(Reading database ... 333843 files and directories currently installed.)
Preparing to unpack libglib2.0-0_2.48.2-0ubuntu4_arm64.deb ...
Unpacking libglib2.0-0:arm64 (2.48.2-0ubuntu4) over (2.48.2-0ubuntu4) ...
Setting up libglib2.0-0:arm64 (2.48.2-0ubuntu4) ...
/usr/lib/aarch64-linux-gnu/glib-2.0/glib-compile-schemas: 2: /usr/lib/aarch64-linux-gnu/glib-2.0/glib-compile-schemas: Syntax error: "&" unexpected
Processing triggers for libc-bin (2.23-0ubuntu10) ...
Processing triggers for libglib2.0-0:amd64 (2.48.2-0ubuntu4) ...

The deb file can't run glib-compile-schemas :(

[sdroege]

Ah you're on some ancient distro. That's indeed a problem then :)

I would suggest taking a look at cerbero for building GStreamer and all its dependencies in an automated way. Alternatively without the dependencies (you have to get them from e.g. your distro then, apt-get build-dep will help), gst-build will be useful (that's meson with all GStreamer modules as meson subprojects). Both should working fine for cross-compilation (I used both for cross-compilation in the past).

Apart from that, overall your problem is a bug in the distro packages of the distro you're using, and the last of those I just fixed today so it will probably take a while until it reaches stable distro releases (you can backport that fix and rebuild your old distro packages I guess).

---

The other problem there is that I believe we're tied to gstreamer-1.8 on the Jetson since we definitely require the various nvidia plugins (Maybe the sources for these are available somewhere though).

Some are, some aren't. For the ones that are, @ystreet has some branches that port them to GStreamer 1.14 if I remember correctly.

[nirbheek]

Both should working fine for cross-compilation (I used both for cross-compilation in the past).

One issue you will face is that Cerbero disables gobject-introspection if we're cross-compiling because gobject-introspection has to compile and run an executable built for the target platform. GIRs and typelibs are necessary to be able to use gstreamer-rs.

However, this can be fixed by installing binfmt support for qemu which will let you run aarch64 apps transparently through qemu. Search the cerbero codebase for variants.gi and introspection.

[sdroege]

gobject-introspection support is not needed for using GStreamer from Rust though, so that's fine

[amotl]

Short addendum: The reason that the arm64 glib library doesn't get installed:

# ERROR: Some arbitrary horrible installation problem on Ubuntu.

The deb file can't run glib-compile-schemas :(

When starting the development of saraswati supported by cross the other day, we found Ubuntu to have rather poor support for multiarch. That's the reason we forked some of @japaric's cross code (thanks for that!) into https://github.com/hiveeyes/saraswati/tree/master/rust/docker in order to build the Docker images based on Debian instead of Ubuntu.

So, you might want to check whether your build environment could be based on Debian stable and some of your problems might just go away.

We also added a little walkthrough describing our setup process which you might enjoy reading at Cross compiling Rust programs using "cross", where you can also find the rationale we based our decision upon:

While Jorge was building the filesystem images for "cross" upon Ubuntu 16.04.4 LTS (Xenial Xerus), we decided to switch to vanilla Debian GNU/Linux 9 (stretch) because Debian is fully supporting the MIPS platform with binary builds, thus relieving us from the pain of having to compile GLib and GStreamer against MIPS from scratch.

Edit: We didn't know of cerbero back then ;].

---

@sdroege and @nirbheek: Thanks for sharing your insights, especially about cerbero which was new to us as we are just starting our journey into the GStreamer ecosystem.

[nirbheek]

@sdroege and @nirbheek: Thanks for sharing your insights, especially about cerbero which was new to us as we are just starting our journey into the GStreamer ecosystem.

Note that you will want to use the 1.14 stable branch and not git master which is a moving target and may change or break.

[amotl]

Note that you will want to use the 1.14 stable branch and not git master which is a moving target and may change or break.

Good to know, thanks again!

[sdroege]

This is now tracked at https://gitlab.freedesktop.org/gstreamer/gstreamer-rs/issues/130