`bpf-linker` panics with `Unable to open LLVM shared lib: [...]/libLLVM-18-rust-1.79.0-nightly.so: file too short`

[javierhonduco]

Hi,

When trying to compile https://github.com/aya-rs/book/tree/1d62046f4addf513ceb29298e38116625e776a3e/examples/xdp-hello cargo xtask build-ebpf panics with Unable to open LLVM shared lib: [...]/libLLVM-18-rust-1.79.0-nightly.so: file too short, see full stack trace below. I am not sure if this could be related to https://github.com/aya-rs/bpf-linker/issues/210, the stack trace contains a similar top error, but it looks a bit different. Thanks!

cc @rdelfin

Full stacktrace

``` error: linking with `bpf-linker` failed: exit status: 101 | = note: LC_ALL="C" PATH="/home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/javierhonduco/.bun/bin:/usr/local/go/bin:/usr/lib/golang/bin/go:/home/javierhonduco/.rbenv/bin:/home/javierhonduco/.rbenv/shims:/home/javierhonduco/.rbenv/bin:/home/javierhonduco/go/bin:/home/javierhonduco/.nvm/versions/node/v16.13.0/bin:/home/javierhonduco/.cargo/bin:/home/javierhonduco/.local/bin:/home/javierhonduco/bin:/home/javierhonduco/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin" VSLANG="1033" "bpf-linker" "--export-symbols" "/tmp/rustcwjYinT/symbols" "/tmp/rustcwjYinT/symbols.o" "/home/javierhonduco/src/book/examples/xdp-hello/xdp-hello-ebpf/../target/bpfel-unknown-none/debug/deps/xdp_hello-10a2003513780c7c.xdp_hello.b94140af8b949ec1-cgu.0.rcgu.o" "-L" "/home/javierhonduco/src/book/examples/xdp-hello/xdp-hello-ebpf/../target/bpfel-unknown-none/debug/deps" "-L" "/home/javierhonduco/src/book/examples/xdp-hello/xdp-hello-ebpf/../target/debug/deps" "-L" "/home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/bpfel-unknown-none/lib" "--cpu" "generic" "-L" "/home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/bpfel-unknown-none/lib" "-o" "/home/javierhonduco/src/book/examples/xdp-hello/xdp-hello-ebpf/../target/bpfel-unknown-none/debug/deps/xdp_hello-10a2003513780c7c" "-O3" "--debug" = note: Unable to open LLVM shared lib: /home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-18-rust-1.79.0-nightly.so: file too short thread 'main' panicked at 'explicit panic', /home/javierhonduco/.cargo/registry/src/github.com-1ecc6299db9ec823/aya-rustc-llvm-proxy-0.4.0/src/lib.rs:75:17 stack backtrace: 0: 0x5599fa576eed - std::backtrace_rs::backtrace::libunwind::trace::h22893a5306c091b4 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/../../backtrace/src/backtrace/libunwind.rs:93:5 1: 0x5599fa576eed - std::backtrace_rs::backtrace::trace_unsynchronized::h29c3bc6f9e91819d at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 2: 0x5599fa576eed - std::sys_common::backtrace::_print_fmt::he497d8a0ec903793 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys_common/backtrace.rs:66:5 3: 0x5599fa576eed - ::fmt::h9c2a9d2774d81873 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys_common/backtrace.rs:45:22 4: 0x5599fa5938fc - core::fmt::write::hba4337c43d992f49 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/fmt/mod.rs:1194:17 5: 0x5599fa5745f1 - std::io::Write::write_fmt::heb73de6e02cfabed at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/io/mod.rs:1655:15 6: 0x5599fa578845 - std::sys_common::backtrace::_print::h63c8b24acdd8e8ce at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys_common/backtrace.rs:48:5 7: 0x5599fa578845 - std::sys_common::backtrace::print::h426700d6240cdcc2 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys_common/backtrace.rs:35:9 8: 0x5599fa578845 - std::panicking::default_hook::{{closure}}::hc9a76eed0b18f82b at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:295:22 9: 0x5599fa5784f9 - std::panicking::default_hook::h2e88d02087fae196 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:314:9 10: 0x5599fa578d92 - std::panicking::rust_panic_with_hook::habfdcc2e90f9fd4c at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:698:17 11: 0x5599fa55ad5b - std::panicking::begin_panic::{{closure}}::h835707b40f64487e 12: 0x5599fa55ad24 - std::sys_common::backtrace::__rust_end_short_backtrace::h79d28ddfc213af0f 13: 0x5599fa4ce17a - std::panicking::begin_panic::h7328ce24b96cb83d 14: 0x5599fa558f01 - std::sync::once::Once::call_once::{{closure}}::h41b21e1b69140d08 15: 0x5599fa4cf12a - std::sync::once::Once::call_inner::h8b7937883f1da40f at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sync/once.rs:434:21 16: 0x5599fa55754e - LLVMInitializeBPFTarget 17: 0x5599fa4f22e0 - bpf_linker::llvm::init::h5dc90320967e0641 18: 0x5599fa4ecdda - bpf_linker::linker::Linker::link::h3f937367f1914f40 19: 0x5599fa4e4f0d - bpf_linker::main::hf636bd6a42f47524 20: 0x5599fa4eaae3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h2cada9d54d4f7eef 21: 0x5599fa4d8f19 - std::rt::lang_start::{{closure}}::h9cbc01e35539e700 22: 0x5599fa56e9fe - core::ops::function::impls:: for &F>::call_once::had4f69b3aefb47a8 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:259:13 23: 0x5599fa56e9fe - std::panicking::try::do_call::hf2ad5355fcafe775 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:492:40 24: 0x5599fa56e9fe - std::panicking::try::h0a63ac363423e61e at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:456:19 25: 0x5599fa56e9fe - std::panic::catch_unwind::h18088edcecb8693a at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panic.rs:137:14 26: 0x5599fa56e9fe - std::rt::lang_start_internal::{{closure}}::ha7dad166dc711761 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/rt.rs:128:48 27: 0x5599fa56e9fe - std::panicking::try::do_call::hda0c61bf3a57d6e6 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:492:40 28: 0x5599fa56e9fe - std::panicking::try::hbc940e68560040a9 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:456:19 29: 0x5599fa56e9fe - std::panic::catch_unwind::haed0df2aeb3fa368 at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panic.rs:137:14 30: 0x5599fa56e9fe - std::rt::lang_start_internal::h9c06694362b5b80c at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/rt.rs:128:20 31: 0x5599fa4e5722 - main 32: 0x7f217c0e0088 - __libc_start_call_main 33: 0x7f217c0e014b - __libc_start_main_impl 34: 0x5599fa4cfa65 - _start 35: 0x0 - ```

Environment info

``` [javierhonduco@fedora xdp-hello]$ file /home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-18-rust-1.79.0-nightly.so /home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-18-rust-1.79.0-nightly.so: ASCII text ``` ``` [javierhonduco@fedora xdp-hello]$ cat /home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-18-rust-1.79.0-nightly.so INPUT(libLLVM.so.18.1-rust-1.79.0-nightly) ``` ``` /home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM.so.18.1-rust-1.79.0-nightly: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped ``` ``` [javierhonduco@fedora xdp-hello]$ ls -lah /home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM.so.18.1-rust-1.79.0-nightly` -rw-r--r--. 1 javierhonduco javierhonduco 139M Apr 25 11:38 /home/javierhonduco/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM.so.18.1-rust-1.79.0-nightly ``` ``` [javierhonduco@fedora xdp-hello]$ rustc --version rustc 1.79.0-nightly (ef8b9dcf2 2024-04-24) ``` ``` [javierhonduco@fedora xdp-hello]$ cat /etc/os-release NAME="Fedora Linux" VERSION="40 (Workstation Edition)" ID=fedora VERSION_ID=40 VERSION_CODENAME="" PLATFORM_ID="platform:f40" PRETTY_NAME="Fedora Linux 40 (Workstation Edition)" ANSI_COLOR="0;38;2;60;110;180" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:40" DEFAULT_HOSTNAME="fedora" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f40/system-administrators-guide/" SUPPORT_URL="https://ask.fedoraproject.org/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=40 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=40 SUPPORT_END=2025-05-13 VARIANT="Workstation Edition" VARIANT_ID=workstation ``` ``` [javierhonduco@fedora xdp-hello]$ bpf-linker --version bpf-linker 0.9.4 ```

[javierhonduco]

bpf-linker 0.9.4 was released 2y ago according to https://crates.io/crates/bpf-linker/versions, I tried installing the latest version, 0.9.13, and it works fine. Not closing the issue just yet because I believe @rdelfin spotted this error with a modern bpf-linker version on his box

[vadorovsky]

Not closing the issue just yet because I believe @rdelfin spotted this error with a modern bpf-linker version on his box

If the similar error still occurs with the newest bpf-linker, it most likely means that Rust nightly is outdated and uses LLVM 18. Make sure that you have both up to date bpf-linker and nightly toolchain.

The core of this issue is that LLVM version used by Rust compiler needs to match the LLVM version used by bpf-linker. So if you are using nightly from this month, you need LLVM 19. If you're using nightly from before they switched to LLVM 19, you need LLVM 18.

But if the issue still persists with both being up to date, please post the full log. :)

[rdelfin]

Thanks. I'll do my best to replicate it and provide more details. It was quite a strange issue when we hit it

[javierhonduco]

Thanks for your reply!

The core of this issue is that LLVM version used by Rust compiler needs to match the LLVM version used by bpf-linker.

That's very good to know, was not aware of this.

[javierhonduco]

@vadorovsky would it be ok to open an issue to add a check to ensure the LLVM version of rustc and bpf-linker matches? I see that rustc prints it with rustc --version --verbose. Let me know!

[Sherlock-Holo]

I also meet this problem, however when I build with --release it disappear

the error output is

error: linking with bpf-linker failed: exit status: 2 | = note: LC_ALL="C" PATH="/home/sherlock/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/sherlock/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/sherlock/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/sherlock/.local/bin:/home/sherlock/.cargo/bin:/home/sherlock/.local/bin:/home/sherlock/.cargo/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/rustup/bin:/home/sherlock/.local/share/JetBrains/Toolbox/scripts" VSLANG="1033" "bpf-linker" "--export-symbols" "/tmp/rustctSZAxE/symbols" "/tmp/rustctSZAxE/symbols.o" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/bpf-5f82aa83bb828146.1wmi8vltjyih6w6vs5hpz3gce.rcgu.o" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/bpf-5f82aa83bb828146.2thvllu9xv7ixzcfne74nxk3b.rcgu.o" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/bpf-5f82aa83bb828146.3514g41f843i7g6elovcoob4q.rcgu.o" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/bpf-5f82aa83bb828146.4qp9v74o4xw7r6ui4znhdnlce.rcgu.o" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/bpf-5f82aa83bb828146.5krpiu9po0gy7s8aqw2ej39k2.rcgu.o" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/libbpf-e10d06608150ea75.rlib" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/libaya_log_ebpf-483ff5ff8c8f7bd4.rlib" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/libaya_log_common-cde91de2d19ed58d.rlib" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/libnum_enum-2b3374f4d0e1099c.rlib" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/libaya_ebpf-79ba6bc20d234ef8.rlib" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/libaya_ebpf_bindings-66840fc57a124bea.rlib" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/libaya_ebpf_cty-b572a482c2d98ac8.rlib" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/librustc_std_workspace_core-3cbf9c5d8705edfa.rlib" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/libcore-5023775ddfe6d40b.rlib" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/libcompiler_builtins-f9e8c717abd72cac.rlib" "--cpu" "generic" "-o" "/home/sherlock/git/wiresquirrel/target/bpfel-unknown-none/debug/deps/bpf-5f82aa83bb828146" "-O0" "--debug" "--btf" = note: unable to open LLVM shared lib /home/sherlock/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-19-rust-1.82.0-nightly.so: /home/sherlock/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-19-rust-1.82.0-nightly.so: file too short error: LLVM issued diagnostic with error severity

I am using nightly rustc

rustc 1.82.0-nightly (0d634185d 2024-08-29) binary: rustc commit-hash: 0d634185dfddefe09047881175f35c65d68dcff1 commit-date: 2024-08-29 host: x86_64-unknown-linux-gnu release: 1.82.0-nightly LLVM version: 19.1.0

that's my .cargo/config.toml file

[build]
target = "bpfel-unknown-none"

[unstable]
build-std = ["core"]

[target.bpfel-unknown-none]
rustflags = ["-C", "debuginfo=2", "-C", "link-arg=--btf"]

bpf-linker version is 0.9.12

[qmonnet]

Hi, for what it's worth, I'm hitting the same thing on my setup:

$ bpf-linker --version           
bpf-linker 0.9.13
$ rustc --version --verbose
rustc 1.83.0-nightly (8d6b88b16 2024-09-11)
binary: rustc
commit-hash: 8d6b88b168e45ee1624699c19443c49665322a91
commit-date: 2024-09-11
host: x86_64-unknown-linux-gnu
release: 1.83.0-nightly
LLVM version: 19.1.0

$ cat /home/qmo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-19-rust-1.83.0-nightly.so
INPUT(libLLVM.so.19.1-rust-1.83.0-nightly)

I get a similar error message:

$ cargo build --target=bpfel-unknown-none -Z build-std=core
[...]
  = note: unable to open LLVM shared lib /home/qmo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-19-rust-1.83.0-nightly.so: /home/qmo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-19-rust-1.83.0-nightly.so: file too short
          ERROR llvm: <unknown>:0:0: in function memmove ptr (ptr, ptr, i64): A call to built-in function 'abort' is not supported.

          ERROR llvm: <unknown>:0:0: in function memmove ptr (ptr, ptr, i64): A call to built-in function 'abort' is not supported.

          Error: LLVM issued diagnostic with error severity

Adding --release fixes the issue

[vadorovsky]

@javierhonduco

@vadorovsky would it be ok to open an issue to add a check to ensure the LLVM version of rustc and bpf-linker matches? I see that rustc prints it with rustc --version --verbose. Let me know!

I'm very sorry for missing this comment! Sure, we could add such check, I think it would be valuable.

[vadorovsky]

@qmonnet

Hi, for what it's worth, I'm hitting the same thing on my setup:

$ bpf-linker --version           
bpf-linker 0.9.13
$ rustc --version --verbose
rustc 1.83.0-nightly (8d6b88b16 2024-09-11)
binary: rustc
commit-hash: 8d6b88b168e45ee1624699c19443c49665322a91
commit-date: 2024-09-11
host: x86_64-unknown-linux-gnu
release: 1.83.0-nightly
LLVM version: 19.1.0

$ cat /home/qmo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-19-rust-1.83.0-nightly.so
INPUT(libLLVM.so.19.1-rust-1.83.0-nightly)

I get a similar error message:

$ cargo build --target=bpfel-unknown-none -Z build-std=core
[...]
  = note: unable to open LLVM shared lib /home/qmo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-19-rust-1.83.0-nightly.so: /home/qmo/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-19-rust-1.83.0-nightly.so: file too short
          ERROR llvm: <unknown>:0:0: in function memmove ptr (ptr, ptr, i64): A call to built-in function 'abort' is not supported.

          ERROR llvm: <unknown>:0:0: in function memmove ptr (ptr, ptr, i64): A call to built-in function 'abort' is not supported.

          Error: LLVM issued diagnostic with error severity

Adding --release fixes the issue

This is a different issue than the original one. I think this one is about a lack of memmove builtin. We fixed it in aya-rs/aya#971, but that fix is not released yet.

To double-check whether that's really an issue, you could use aya-ebpf from git (aya-ebpf = { git = "https://github.com/aya-rs/aya", branch = "main" } in your Cargo.toml).

Or you could try adding the following code in your eBPF program code:

#[no_mangle]
pub unsafe extern "C" fn memmove(dest: *mut u8, src: *mut u8, n: usize) {
    let delta = (dest as usize).wrapping_sub(src as usize);
    if delta >= n {
        // We can copy forwards because either dest is far enough ahead of src,
        // or src is ahead of dest (and delta overflowed).
        copy_forward(dest, src, n);
    } else {
        copy_backward(dest, src, n);
    }
}

#[inline(always)]
unsafe fn copy_forward(dest: *mut u8, src: *mut u8, n: usize) {
    for i in 0..n {
        *dest.add(i) = *src.add(i);
    }
}

#[inline(always)]
unsafe fn copy_backward(dest: *mut u8, src: *mut u8, n: usize) {
    for i in (0..n).rev() {
        *dest.add(i) = *src.add(i);
    }
}

Let me know whether that helps.

If it doesn't, an another option might be that you have code which can panic (unwrap, panic! etc.). Panics are not supported in Aya for now and all errors have to be handled explicitly.

[qmonnet]

Apologies for mixing up two different issues. Yes, it does work correctly in my case with the current HEAD from main, thank you Michal!

[vadorovsky]

@qmonnet No worries, happy to hear that it works for you now! :slightly_smiling_face:

@javierhonduco Are you still facing issues on your side?

[javierhonduco]

cc @rdelfin 👀

[vadorovsky]

I will close this issue for now, feel free to re-open if it still occurs.

The original issue reported was fixed by #214, the release was already made (0.9.13).

And @qmonnet's issue is fixed by aya-rs/aya#971, which is not released yet.