search

rust-cross / cargo-zigbuild

Compile Cargo project with zig as linker
MIT License
1.43k stars 51 forks source link

Cross compile on aarch64 from MacOS/darwin to aarch64-unknown-linux-musl is failing with undefined symbol: readdir64 #179

Closed reubenmiller closed 11 months ago

reubenmiller commented 11 months ago

When building with the newly release cargo-zigbuild version 0.17.2, the build fails with the following errors. From a quick inspection it looks to be related to https://github.com/rust-cross/cargo-zigbuild/pull/172 (though it is just a guess). Reverting back to cargo-zigbuild 0.17.1 resolves the issue.

The error that I get when cross compiling from MacOS M1 to aarch64-unknown-linux-musl is:

error: linking with `/Users/reubenmiller/Library/Caches/cargo-zigbuild/0.17.2/zigcc-aarch64-unknown-linux-musl.sh` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin/self-contained:/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/tests/RobotFramework/.venv/bin:/Users/reubenmiller/.nvm/versions/node/v16.20.0/bin:/Users/reubenmiller/homebrew/opt/gnu-tar/libexec/gnubin:/Users/reubenmiller/go/bin:/Users/reubenmiller/.gem/ruby/2.6.0/bin:/Users/reubenmiller/.local/bin:/Users/reubenmiller/homebrew/bin:/Users/reubenmiller/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/reubenmiller/.nvm/versions/node/v16.20.0/bin:/Users/reubenmiller/homebrew/opt/gnu-tar/libexec/gnubin:/Users/reubenmiller/go/bin:/Users/reubenmiller/.gem/ruby/2.6.0/bin:/Users/reubenmiller/.local/bin:/Users/reubenmiller/homebrew/bin:/Users/reubenmiller/homebrew/sbin:/Users/reubenmiller/.cargo/bin" VSLANG="1033" "/Users/reubenmiller/Library/Caches/cargo-zigbuild/0.17.2/zigcc-aarch64-unknown-linux-musl.sh" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crti.o" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtbegin.o" "/var/folders/wn/06nj6dqj07db9tbsh_094wnr0000gn/T/rustckaNAp4/symbols.o" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/deps/sawtooth_publisher-8617a97078f8b92f.sawtooth_publisher.9260d1d0-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/deps" "-L" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/release/deps" "-L" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/build/ring-0931aa783d916770/out" "-L" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/var/folders/wn/06nj6dqj07db9tbsh_094wnr0000gn/T/rustckaNAp4/libring-751c867e148decbc.rlib" "-lunwind" "-lc" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-a546a2beee625b76.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-nostartfiles" "-L" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib" "-L" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained" "-o" "/Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/deps/sawtooth_publisher-8617a97078f8b92f" "-Wl,--gc-sections" "-static" "-no-pie" "-Wl,-z,relro,-z,now" "-Wl,--strip-all" "-nodefaultlibs" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtend.o" "/Users/reubenmiller/.rustup/toolchains/1.70-aarch64-apple-darwin/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtn.o"
  = note: ld.lld: error: undefined symbol: readdir64
          >>> referenced by sawtooth_publisher.9260d1d0-cgu.0
          >>>               /Users/reubenmiller/dev/projects/tedge/tmp/thin-edge.io/target/aarch64-unknown-linux-musl/release/deps/sawtooth_publisher-8617a97078f8b92f.sawtooth_publisher.9260d1d0-cgu.0.rcgu.o:(OUTLINED_FUNCTION_5589)

Version information

$ rustc --version
rustc 1.70.0 (90c541806 2023-05-31

$ cargo --version
cargo 1.70.0 (ec8a8a0ca 2023-04-25)

$ python3 -m ziglang version
0.11.0

$ cargo-zigbuild --version
cargo-zigbuild 0.17.1
messense commented 11 months ago

cc @haohaolee Looks like the weak symbol list isn't complete?

https://github.com/search?q=repo%3Abminor%2Fmusl%20_LARGEFILE64_SOURCE&type=code

messense commented 11 months ago

@reubenmiller IMO it should be easy to fix by adding more LFS64 functions to https://github.com/rust-cross/cargo-zigbuild/blob/846edea5911c82ebe1d2279854ae60b4f9268a1c/src/linux/mod.rs#L22-L27

Happy to accept a PR to do that.

reubenmiller commented 11 months ago

I can confirm that the build experienced will be fixed once https://github.com/rust-cross/cargo-zigbuild/pull/180 is merged.

haohaolee commented 11 months ago

I wanted to be conservative so I tried to modify as few symbols as possible in https://github.com/rust-cross/cargo-zigbuild/pull/172, I was afraid that too many adds lead to other issues.

Hi @reubenmiller does your project rely on libc crate? as far as I know, some symbols like readdir64 are referenced in libc crate, and the latest libc crate has already fixed this issue.

Please let me know if this is the case, thanks

haohaolee commented 11 months ago

I have managed to figure out this, I did miss some symbols in my PR https://github.com/rust-cross/cargo-zigbuild/pull/172:

nm ~/.rustup/toolchains/1.71.1-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-493936ddc336b3cf.rlib | grep "64$"
nm: lib.rmeta: no symbols
                 U fstat64
                 U fstatat64
                 U ftruncate64
  ......
                 U lseek64
                 U lstat64
                 U open64
                 U pread64
                 U pwrite64
                 U readdir64
                 U stat64

At least these are the symbols needed. And for 1.72.0, aka stable:

nm ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-aa8203b3ba116fbe.rlib | grep "64$"
nm: lib.rmeta: no symbols
0000000000000000 r GCC_except_table164
   ......

It would be better to have a test for this regression.

reubenmiller commented 11 months ago

I wanted to be conservative so I tried to modify as few symbols as possible in #172, I was afraid that too many adds lead to other issues.

Hi @reubenmiller does your project rely on libc crate? as far as I know, some symbols like readdir64 are referenced in libc crate, and the latest libc crate has already fixed this issue.

Please let me know if this is the case, thanks

No we don't use anything from the libc crate, we use musl builds for portability reasons.

haohaolee commented 11 months ago

@reubenmiller No worries. Ignore me, I was asking a wrong question. I have checked your build and I think I have a better understanding now, this issue has nothing to do with the libc crate.

In short, you need to satisfy those undefined symbols in std, e.g. U readdir64

ps: musl is a kinda libc, and your project does depend on libc crate if you check the Cargo.lock