Unable to link a binary with wasmtime and musl

pimeys commented 2 days ago

Test Case

I'm sorry I cannot find an isolated example for you to test with a small dependency tree, but in our repo, when we updated wasmtime from 21.0.1 to 22.0.0 we suddenly started having issues linking the binary when compiling against musl target.

Here's the branch:

https://github.com/grafbase/grafbase/pull/1840

And the error we see when linking:

  = note: /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(e_pow.o): warning: relocation against `_dl_x86_cpu_features' in read-only section `.text'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_ceil.o): in function `__ceil_ifunc':
          (.text+0x6): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_trunc.o): in function `__trunc_ifunc':
          (.text+0x6): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_fma.o): in function `__fma_ifunc':
          (.text+0xafd): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: (.text+0xb06): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(w_fmod.o): in function `__fmod':
          (.text+0x41): undefined reference to `errno'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(w_pow.o): in function `__pow':
          (.text+0x75): undefined reference to `errno'
          /usr/bin/ld: (.text+0xab): undefined reference to `errno'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(s_truncf.o): in function `__truncf_ifunc':
          (.text+0x6): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(math_err.o): in function `with_errno':
          (.text+0x3): undefined reference to `errno'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(math_err.o): in function `with_errno.constprop.0':
          (.text+0x13): undefined reference to `errno'
          /usr/bin/ld: /usr/lib/x86_64-linux-gnu/libm-2.35.a(e_pow.o): in function `__ieee754_pow_ifunc':
          (.text+0x746): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: (.text+0x74f): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: (.text+0x75f): undefined reference to `_dl_x86_cpu_features'
          /usr/bin/ld: read-only segment has dynamic relocations
          collect2: error: ld returned 1 exit status

We can currently work with the previous version, but this issue prevents us from updating to the latest runtime.

Steps to Reproduce

fetch the branch linked in the previous chapter
Make sure to have musl target installed from rustup
Install musl-gcc if needed
cargo build -p grafbase-gateway --release --target x86_64-unknown-linux-musl

Expected Results

The project compiles correctly into a working binary.

Actual Results

Linking fails as explained in the first chapter.

Versions and Environment

Wasmtime version or commit: 22.0.0

Operating system: NixOS unstable and Debian Bookworm

Architecture: x86_64-unknown-linux-musl

Extra Info

This is most probably some issue with wasmtime and some other crate together. I could not reduce a nice and clean repro as a new project. Been trying to get that the whole afternoon... But something definitely broke in the latest update of wasmtime.

bjorn3 commented 2 days ago

For one reason or another the linker is using glibc's version of libm rather than musl's version of libm. I'm not sure why that happens,. In any case you shouldn't need musl installed on the host system, nor use musl-gcc. Rustc already ships with a copy of musl's libc.a which includes libm.

pimeys commented 2 days ago

This might be it. I installed a clean debian container to test this out and some dependency didn't build due to missing musl-gcc... I will get back to this when I'm on my computer again.

pimeys commented 1 day ago

So, this is what I found out:

wasmtime depends on ittapi, which depends on ittapi-sys
ittapi-sys depends on cc crate, and fails to compile on a system without musl-gcc
if installing musl-tools, the crate compiles but with version 22 of wasmtime the linking stage fails to the error described in the issue

I wonder how common these kind of issues are with musl? We do have several sys dependencies which contain C code and which compile just fine against musl. But why did the wasmtime 22 update break the linking?

pimeys commented 1 day ago

I did a git-bisect today trying to find which commit broke the build and I think I found it:

https://github.com/bytecodealliance/wasmtime/commit/cc93e8f7f0476c89e1a2ca3a27518e0a7c4177aa#diff-0edccdae9479a6e6d11da47ad26d419b88045e14c5a6086ce7cdd8e236132ed4R38-R41

Ping @alexcrichton

Removing these lines from build.rs fixes the issue. But this probably breaks something else...

bytecodealliance / wasmtime