LTO causes sysroot to be unlinkable by non-LLVM toolchains

[rbtying]

I'm using cargo xbuild to compile Rust binaries for the ESP32 via https://github.com/MabezDev/rust-xtensa, and then linking them via the Espressif xtensa-esp32-elf-ld, which is a GNU crosstool-NG linker that doesn't understand LLVM bitcode / LTO output

Is it possible to disable LTO by default for the sysroot, or make it a flag that can be passed in?

Reverting to 0.5.29 resolves the issue, so I imagine this is caused by the fix for #69 ?

[phil-opp]

Which version of cargo-xbuild are you using? We enabled LTO in version 0.5.30, but reverted that change again in version 0.5.31.

[phil-opp]

I just created a pull request to use a different flag for the sysroot build in https://github.com/rust-osdev/cargo-xbuild/pull/73. Could you try whether this new version fixes your issue. You can install it through:

cargo install cargo-xbuild --git https://github.com/rust-osdev/cargo-xbuild.git --branch embed-bitcode  --debug --force

[rbtying]

@phil-opp the embed-bitcode flag doesn't seem to be understood by my rustc branch (based off of llvm-9)

error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `rustc - --crate-name ___ --print=file-names -Cembed-bitcode=yes --target xtensa-esp32-none-elf --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit code: 1)
--- stderr
error: unknown codegen option: `embed-bitcode`

You can reproduce this issue by setting up a crossbuild docker image from https://github.com/rbtying/rust-xtensa-docker, and then opening a bash session in the container with

docker run --rm --mount type=bind,source=$(pwd),target=/project -it rbtying/esp-crossbuild-env-user /bin/bash

running

cargo install cargo-xbuild --git https://github.com/rust-osdev/cargo-xbuild.git --branch embed-bitcode  --debug --force

and calling

cargo +xtensa xbuild --target xtensa-esp32-none-elf --release

on an empty staticlib crate with LTO turned off, or this crate: https://github.com/rbtying/esp32-hello/tree/master/main

To determine if the resulting file is valid or not, it should suffice to run nm on the resulting .a file -- if it's not, GNU nm should say so.

[phil-opp]

What is your rustc version? As far as I know, the -Cembed-bitcode flag was named differently until recently, so perhaps that's the issue.

Have you tried compiling with the most recent crates.io release of cargo-xbuild (version 0.5.31)? It should no longer enable lto.

[rbtying]

With 0.5.31, I still get the error:

nm: alloc-622d879c6be42eb3.alloc.8vxpzbbw-cgu.0.rcgu.o: file format not recognized
nm: compiler_builtins-81223a7640e2dd04.compiler_builtins.b2sjtz44-cgu.0.rcgu.o: file format not recognized
nm: rustc_std_workspace_core-a7493121c62fcf4b.rustc_std_workspace_core.zm2lj6do-cgu.0.rcgu.o: file format not recognized
nm: core-d2bbeb4c215585a6.core.8icfffy5-cgu.0.rcgu.o: file format not recognized

The rustc version is forked from 209b2be09fcaff937480d1fbbe8b31646e361c7a, which is from April of this year? Somewhere around 1.42-1.44, I think.

That said, it would be unfortunate if cargo-xbuild only works with the most recent rustc versions, since it's not always easy to update toolchains to catch up.

[phil-opp]

Thanks for testing! Did you run cargo clean before cargo xbuild to ensure that everything is rebuilt?

That said, it would be unfortunate if cargo-xbuild only works with the most recent rustc versions, since it's not always easy to update toolchains to catch up.

We would love to support older rustc versions too if it would be easily possible. Unfortunately, there are sometimes breaking changes in nightly rustc itself that force us to change the way the sysroot is built to unbreak users of newer nightlies. The new version will then probably not work on older nightlies.

While it would be possible in theory to detect the rustc version and then act accordingly to the reported version, it would be an enormous task to keep support for all older nighlies. I don't have the necessary time for that, so I think it's a reasonable policy to just support the latest nightly. Users that pinned older rustc versions can just pin older cargo-xbuild versions as well to keep things working.

---

As a side note, have you tried using cargo's own -Zbuild-std flag instead of cargo-xbuild? It already works for most use cases and is always in sync with the respective nightly.

[rbtying]

Thanks for looking into this! I did run cargo clean prior to cargo xbuild with version 0.5.31 as released on crates.io

I also rebased the xtensa patches onto the latest rustc from earlier today, and tried to use the embed-bitcode flag. It seems to work! (or at least, it links).

We would love to support older rustc versions too if it would be easily possible. Unfortunately, there are sometimes breaking changes in nightly rustc itself that force us to change the way the sysroot is built to unbreak users of newer nightlies. The new version will then probably not work on older nightlies.

I think it's reasonable not to support all nightlies into the past. Perhaps worthwhile to document when backwards-incompatible changes are made? In this case, the nightly is from a month or so ago, so the new flag is a very recent change.

As a side note, have you tried using cargo's own -Zbuild-std flag instead of cargo-xbuild? It already works for most use cases and is always in sync with the respective nightly.

I didn't know this existed! I'm not sure how to use -Zbuild-std with a local build of the Rust compiler, though: it seems to assume that you can install various components from rustup? Will look more into this, since reducing the set of dependencies seems useful.

[lights0123]

On my embedded target and compiling with lto = true like I normally do, I receive the error:

error: linking with `nspire-gcc` failed: exit code: 1
  |
  = note: "nspire-gcc" "-L" "/home/benschattinger/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/armv5te-nspire-eabi/lib" "/home/benschattinger/Documents/Projects/Calculator/example-nspire/target/armv5te-nspire-eabi/release/deps/async_demo-f656ddd305f9abb2.async_demo.1vz56qmh-cgu.4.rcgu.o" "-o" "/home/benschattinger/Documents/Projects/Calculator/example-nspire/target/armv5te-nspire-eabi/release/deps/async_demo-f656ddd305f9abb2" "-Wl,--gc-sections" "-nodefaultlibs" "-L" "/home/benschattinger/Documents/Projects/Calculator/example-nspire/target/armv5te-nspire-eabi/release/deps" "-L" "/home/benschattinger/Documents/Projects/Calculator/example-nspire/target/release/deps" "-L" "/home/benschattinger/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/armv5te-nspire-eabi/lib" "-Wl,--start-group" "-Wl,--end-group" "-Wl,-Bstatic" "/home/benschattinger/Documents/Projects/Calculator/example-nspire/target/armv5te-nspire-eabi/release/deps/libcompiler_builtins-8357cebf43967b18.rlib" "-Wl,-Bdynamic" "-Wl,--allow-multiple-definition"
  = note: arm-none-eabi-ld: /home/benschattinger/Documents/Projects/Calculator/example-nspire/target/armv5te-nspire-eabi/release/deps/libcompiler_builtins-8357cebf43967b18.rlib: error adding symbols: file format not recognized
          collect2: error: ld returned 1 exit status

This occurs both on v0.5.31 and embed-bitcode. It seems to be instructing a gcc-style linker to link an rlib, which doesn't work. I receive the same error using -Z build-std=core,alloc.

[zhaofengli]

Can confirm that @phil-opp's #73 does produce binaries that GNU ld understands.

[phil-opp]

@rbtying @zhaofengli Thanks for testing! I will merge the PR then.

@lights0123

I receive the same error using -Z build-std=core,alloc.

This seems like a cargo problem then. I see that you already commented on https://github.com/rust-lang/cargo/issues/8239, so let's hope that this gets resolved soon!

@rbtying

Perhaps worthwhile to document when backwards-incompatible changes are made? In this case, the nightly is from a month or so ago, so the new flag is a very recent change.

I agree that this would be a good idea, however it still requires a lot of work (e.g. testing new versions with many different nightlies, closely following the rustc/cargo development to know when new flags etc were introduced, etc). I'm open to pull requests that add such documentation, but I won't have the time to do it myself unfortunately.

I didn't know this existed! I'm not sure how to use -Zbuild-std with a local build of the Rust compiler, though: it seems to assume that you can install various components from rustup? Will look more into this, since reducing the set of dependencies seems useful.

I'm not sure about that either. I think the location of the Rust source code is detected by running rustc --print=sysroot, so if you can get this to point to your local Rust repo it might work.

[phil-opp]

Closing, as https://github.com/rust-lang/cargo/issues/8239 was fixed.