cargo-bpf build errors on 64-bit ARM host

[cheeseprocedure]

Hello,

I'm running into the following issue when attempting to install cargo-bpf on a 64-bit ARM VM. I think the issue is unique to ARM, as I can't reproduce it in a 64-bit Intel guest:

# cargo install cargo-bpf --no-default-features --features=llvm13,command-line
# ...
error[E0432]: unresolved import `super::__va_list_tag`
  --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/bpf-sys-2.2.0/src/type_gen.rs:22:5
   |
22 |     __va_list_tag, btf, btf__free, btf__get_nr_types, btf__name_by_offset, btf__parse_elf,
   |     ^^^^^^^^^^^^^ no `__va_list_tag` in the root

error[E0308]: mismatched types
   --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/bpf-sys-2.2.0/src/type_gen.rs:186:22
    |
186 |                 Some(vdprintf_wrapper),
    |                      ^^^^^^^^^^^^^^^^ expected array `[u64; 4]`, found *-ptr
    |
    = note: expected fn pointer `unsafe extern "C" fn(_, _, [u64; 4])`
                  found fn item `unsafe extern "C" fn(_, _, *mut [type error]) {vdprintf_wrapper}`

Some errors have detailed explanations: E0308, E0432.
For more information about an error, try `rustc --explain E0308`.
error: could not compile `bpf-sys` due to 2 previous errors
warning: build failed, waiting for other jobs to finish...
error: failed to compile `cargo-bpf v2.2.0`, intermediate artifacts can be found at `/tmp/cargo-installheCcej`

Caused by:
  build failed

I believe I have all the necessary build pre-reqs in place. System/tooling versions:

root@ebpflab1:~# uname -a
Linux ebpflab1 5.10.0-9-arm64 #1 SMP Debian 5.10.70-1 (2021-09-30) aarch64 GNU/Linux

# LLVM installed from apt.llvm.org repos
root@ebpflab1:~# llvm-config-13 --version
13.0.1

root@ebpflab1:~# rustc -vV
rustc 1.56.1 (59eed8a2a 2021-11-01)
binary: rustc
commit-hash: 59eed8a2aac0230a8b53e89d4e99d55912ba6b35
commit-date: 2021-11-01
host: aarch64-unknown-linux-gnu
release: 1.56.1
LLVM version: 13.0.0

Thank you!

[rhdxmr]

@cheeseprocedure It seems that you use Debian 11, right? And did you also use Debian 11 with intel CPU?

[cheeseprocedure]

Correct. Both environments are Debian 11 with the latest 5.10 kernel packages.

[rhdxmr]

@cheeseprocedure I started to look into this issue. But I am not familiar with arm environment it will take time. Thanks for filing this issue.

[cheeseprocedure]

No problem - thank you for looking into it. If there’s further information I can provide please let me know!

[rhdxmr]

Adding aarch64 build-test docker images: https://github.com/foniod/build-images/pull/5

This issue will be resolved soon by another upcoming PR after new aarch64 build-test images are ready.

[ihciah]

I met the same problem compiling on aarch64. Then I investigate the generated code and found the problem.

The __va_list_tag should be __gnuc_list_tag on aarch64. I patched it and it works. Here is my branch https://github.com/ihciah/redbpf/tree/fix-aarch64 , but I don't know if my fix is proper(@rhdxmr if you think it's ok I will submit a PR), I don't know when to use __gnuc_list_tag and when to use the other one.

And also, I think maybe the functions that start with double slash should not be depended directly? If the wrapper can be implemented in a better way, it can be removed.

[rhdxmr]

Hi @ihciah ​

I think your fix is proper. Actually while I am developing aarch64 build-test docker images, I also have been fixing this issue and my solution is almost the same with yours.

At first I thought that using va_list in Rust is not a good choice so I considered to write small C code that wraps vdprintf and make it as static library. And then substitute vdprintf_wrapper with the new C code. But that is somewhat complicated than just using va_list so I didn't choose that.

I had tested that the same error occur in aarch64 Debian11, Ubuntu 21.04, Fedora 35 and Alpine 3.15. And all build errors are solved by substituting __va_list_tag with __gnu_va_list.

So please send a PR and I'll merge it.

ps. I read an article about your monoio in Hacker News few days ago. It is a novel design to me and it makes user code simple. I think this is brilliant!

[rhdxmr]

@cheeseprocedure Thanks to @ihciah #245 solves this issue. So I am closing this issue.