Problem about LIBBPF_SYS_LIBRARY_PATH and build.rs

[kxxt]

When cross-compiling an application that uses libbpf-sys in build.rs(Usually via libbpf-cargo).

If LIBBPF_SYS_LIBRARY_PATH is set to a path that contains the libelf for the target architecture(let's say, aarch64),

then while building the build.rs of this application, we are actually required to build libbpf-sys for the build architecture(x86_64) because the build.rs script needs to link libbpf-sys. But in the build.rs of libbpf-sys, we can only know that we are building for aarch64 target and we are linking aarch64 libelf from LIBBPF_SYS_LIBRARY_PATH into x86_64 libbpf-sys rlib.

static:

 = note: rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_version.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_hash.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_error.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_fill.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_begin.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_next.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_rand.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_end.o) is incompatible with elf64-x86-64

dynamic:

  = note: rust-lld: error: /home/kxxt/repos/tracexec/3rdparty/aarch64/usr/lib/aarch64-linux-gnu/libelf.so is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/3rdparty/aarch64/usr/lib/aarch64-linux-gnu/libz.so is incompatible with elf64-x86-64
          collect2: error: ld returned 1 exit status

This almost makes LIBBPF_SYS_LIBRARY_PATH useless unless libbpf-sys is only used as a dependency but not a build dependency at the same time.

[danielocfb]

Would be best if you could share a sample project and build invocation. Perhaps we have to introduce a variant of this variable to be used in a cross compilation context?

[kxxt]

Would be best if you could share a sample project and build invocation.

I have made a minimal reproduction repo based on the capable example: https://github.com/kxxt/libbpf-sys-106-repro

Running the build-aarch64.sh script should reproduce this bug.

Perhaps we have to introduce a variant of this variable to be used in a cross compilation context?

But the problem is how could we determine the cross compilation context?

[danielocfb]

I have made a minimal reproduction repo based on the capable example: https://github.com/kxxt/libbpf-sys-106-repro

Okay, thanks for this, that's tremendously useful. I finally got the error reproduced.

But the problem is how could we determine the cross compilation context?

I'd think we will be cross-compiling if CARGO_CFG_TARGET_ARCH != std::env::consts::ARCH.

[danielocfb]

Yeah, so if I adjust your script as follows:

--- build-aarch64.sh
+++ build-aarch64.sh
@@ -1,6 +1,8 @@
 #!/bin/sh

-export LIBBPF_SYS_LIBRARY_PATH="$(realpath 3rdparty/aarch64/usr/lib/aarch64-linux-gnu)"
-export LIBBPF_SYS_EXTRA_CFLAGS="-I $(realpath 3rdparty/aarch64/usr/include)"
+export LIBBPF_SYS_LIBRARY_PATH_x86_64="$(realpath 3rdparty/x86_64/usr/lib/x86_64-linux-gnu)"
+export LIBBPF_SYS_EXTRA_CFLAGS_x86_64="-I $(realpath 3rdparty/x86_64/usr/include)"
+export LIBBPF_SYS_LIBRARY_PATH_aarch64="$(realpath 3rdparty/aarch64/usr/lib/aarch64-linux-gnu)"
+export LIBBPF_SYS_EXTRA_CFLAGS_aarch64="-I $(realpath 3rdparty/aarch64/usr/include)"

 cargo build --target aarch64-unknown-linux-gnu -F static

and the libbpf-sys build script like this:

--- build.rs
+++ build.rs
@@ -111,6 +111,7 @@ fn pkg_check(pkg: &str) {

 fn main() {
     let src_dir = path::PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap());
+    let arch = env::var("CARGO_CFG_TARGET_ARCH").unwrap();

     generate_bindings(src_dir.clone());

@@ -156,8 +157,9 @@ fn main() {
             "a C compiler is required to compile libbpf-sys using the vendored copy of libbpf",
         );
         let mut cflags = compiler.cflags_env();
-        println!("cargo:rerun-if-env-changed=LIBBPF_SYS_EXTRA_CFLAGS");
-        if let Some(extra_cflags) = env::var_os("LIBBPF_SYS_EXTRA_CFLAGS") {
+        let sys_cflags_var = format!("LIBBPF_SYS_EXTRA_CFLAGS_{arch}");
+        println!("cargo:rerun-if-env-changed={sys_cflags_var}");
+        if let Some(extra_cflags) = env::var_os(sys_cflags_var) {
             cflags.push(" ");
             cflags.push(extra_cflags);
         }
@@ -198,8 +200,9 @@ fn main() {
     );
     println!("cargo:include={}/include", out_dir.to_string_lossy());

-    println!("cargo:rerun-if-env-changed=LIBBPF_SYS_LIBRARY_PATH");
-    if let Ok(lib_path) = env::var("LIBBPF_SYS_LIBRARY_PATH") {
+    let sys_path_var = format!("LIBBPF_SYS_LIBRARY_PATH_{arch}");
+    println!("cargo:rerun-if-env-changed={sys_path_var}");
+    if let Ok(lib_path) = env::var(sys_path_var) {
         for path in lib_path.split(':') {
             if !path.is_empty() {
                 println!("cargo:rustc-link-search=native={}", path);

Then I get seemingly further:

  = note: rust-lld: error: undefined symbol: ZSTD_createCCtx
          >>> referenced by elf_compress.o:(__libelf_compress) in archive /tmp/libbpf-sys-106-repro/target/debug/deps/liblibbpf_sys-184db90333a73bf9.rlib

The error seems to be caused by the libraries you vendored in the example project being linked against some elfutils built with zstd support, would be my reading of it. So it's not really relevant here, I think, though it does mask any potentially successful build.

[danielocfb]

Likely the next problem we hit will be that, in the above scenario, the skeleton will be created for x86, it seems. Which makes sense, because the build script is always run on the host. But I am not sure it's what you'd want. I suspect we'd need to adjust vmlinux path or something to make that work as well (though I don't know whether that will be enough).

[kxxt]

Likely the next problem we hit will be that, in the above scenario, the skeleton will be created for x86, it seems. Which makes sense, because the build script is always run on the host. But I am not sure it's what you'd want. I suspect we'd need to adjust vmlinux path or something to make that work as well (though I don't know whether that will be enough).

I don't think that is a problem. The build script runs on the host, but cargo passes correct CARGO_CFG_TARGET_ARCH to it so the correct header is used.

[danielocfb]

Have you gotten it to work end-to-end, @kxxt ? If so, feel free to open a pull request.

[kxxt]

Have you gotten it to work end-to-end, @kxxt ? If so, feel free to open a pull request.

No. I didn't try to get it to work. I initially tried to use this variable but find it not working, then switched to what the libbpf-rs CI is doing. But I think it is worth reporting that this variable doesn't work as intended for projects that uses libbpf-cargo or depending on libbpf-sys in any way in build.rs. It's the beginning of a new semester now and I am too busy to work on a solution that I don't actually use.