search

rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
97.61k stars 12.62k forks source link

no_std + liballoc can not compile with os based toolchain caused by eh_personality required #106864

Open gngshn opened 1 year ago

gngshn commented 1 year ago

Update at 2023-01-20

The previous test that met the expected conditions was actually just a case of not testing enough to get the linker's GC optimizations out of the way. The problem is now updated to the fact that in the case of default_alloc_error_handlermerged, the OS-based toolchain cannot compile the no_std + liballoc crate without using the unstable build-std feature, and the error shows up as a missing rust_eh_ personality dependency. Using OS-based toolchain to compile no_std + liballoc is intended to compile small binaries for tiny embedded linux.

I tried this code:

#![no_std]
#![no_main]

extern crate alloc;

use alloc::{boxed::Box, vec};
use libc_alloc::LibcAlloc;
use libc_print::std_name::println;

#[global_allocator]
static GLOBAL_ALLOC: LibcAlloc = LibcAlloc;

#[panic_handler]
fn panic(_: &core::panic::PanicInfo) -> ! {
    unsafe {
        libc::abort();
    }
}

#[no_mangle]
pub extern "C" fn main() -> i32 {
    println!("hello world!");
    let test1 = vec![1, 2, 3];
    for v in test1 {
        println!("{v}")
    }
    let test2 = Box::new(10);
    println!("{test2}");
    0
}

I expected to see this happen: This code build and run perfectly with panic=abort and non nightly features in nightly rust, and when rust 1.68 is comming, we can build this in stable rust. We use no_std + liballoc to build small linux app in embedded linux, in production, we need stable rust.

Instead, this happened: The latest rust nightly can not build this code again(nightly-2022-12-29 is okay), it failed with this link error:

"_rust_eh_personality", referenced from:
                core::panicking::panic_nounwind_fmt::hdc7b70c1a43a8935 in libcore-d0a8c087921eb265.rlib(core-d0a8c087921eb265.core.5d748034-cgu.0.rcgu.o)

this seem caused by #106045

Meta

rustc --version --verbose:

rustc 1.68.0-nightly (0b90256ad 2023-01-13)
Backtrace

``` $ RUST_BACKTRACE=1 cargo build Compiling libc v0.2.139 Compiling libc_alloc v1.0.4 Compiling libc-print v0.1.20 Compiling core_alloc v0.1.0 (/Users/gngshn/develop/rust/core_alloc) error: linking with `cc` failed: exit status: 1 | = note: LC_ALL="C" PATH="/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/bin:/usr/local/opt/coreutils/libexec/gnubin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/gngshn/.cargo/bin:/usr/local/sbin:/Users/gngshn/.local/bin:/Users/gngshn/.local/go/bin:/usr/local/opt/fzf/bin" VSLANG="1033" ZERO_AR_DATE="1" "cc" "-arch" "x86_64" "-m64" "/var/folders/q6/qjg_84cs7m33457bzpwyj89c0000gn/T/rustc8tgW18/symbols.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1e17zqd3jy504d2a.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1eijle777i9sq0pk.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1fe54lcf0ayzfo9i.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1gkto627j4tsu01c.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1krvpfd8926axeg1.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1regfi20mw4f4nr0.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.1ymh0vbbid03fxd7.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.20xrjwb4wnpodfwk.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.236beanv78eknhkk.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2604lu66786wcvm8.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2e5d7pgn1zhe8xd1.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2jkx96ov7p0k0svd.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2mziia6eks3r7iri.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.2qq6ma07ep3fkrie.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3bavlu7c0u97zdy1.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3gw933f5fvrbyrdu.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3nynhhrqjy01lq2w.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3qhxpnb40mit3m4e.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3u938251jwm9ry7f.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.3ze778jtnc4j14s2.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4415dzizekl18aax.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.45ocggxfjsa6x3p.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4ddbu8w29d6lkeu5.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4ijt1knocvq1kql3.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4rlcr2h5yovnam8y.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.4z4w04hvu0o9i9c6.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.5517fkyk2vfortv2.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.jfxexxow8wjy8dn.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.qqacfrqsjnmg87k.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.wk4sxdt5nszlnfu.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.zgclkfsy9t3hh6v.rcgu.o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4.16he8t3edhiz7wic.rcgu.o" "-L" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps" "-L" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/liblibc_print-f9cd8051d3145ecd.rlib" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/liblibc-064686b452655b71.rlib" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/liblibc_alloc-0c85c4bf859b14f8.rlib" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc-c6fc9abf4e398cbb.rlib" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-1bf2f523c1bee03d.rlib" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcore-d0a8c087921eb265.rlib" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-19bcb300462f8eea.rlib" "-lc" "-lm" "-liconv" "-L" "/Users/gngshn/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "-o" "/Users/gngshn/develop/rust/core_alloc/target/debug/deps/core_alloc-0238da2e10db5da4" "-Wl,-dead_strip" "-nodefaultlibs" = note: Undefined symbols for architecture x86_64: "_rust_eh_personality", referenced from: core::panicking::panic_nounwind_fmt::hdc7b70c1a43a8935 in libcore-d0a8c087921eb265.rlib(core-d0a8c087921eb265.core.5d748034-cgu.0.rcgu.o) ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) error: could not compile `core_alloc` due to previous error ```

gngshn commented 1 year ago

Hi @nbdd0121 I have filed a new issue

cc @RalfJung @Amanieu

nbdd0121 commented 1 year ago

Here's one potential solution for this:

This solves the issue here because:

This approach however does add some complexity to the codegen part.

Any thoughts, @Amanieu?

CAD97 commented 1 year ago

Alternatively (but likely of similar complexity), would it be possible to have panic_nounwind_fmt/#[rustc_nounwind] "just" not introduce a dependency on _rust_eh_personality? Since #[rustc_nounwind] wants to catch and immediately abort on any unwind, it doesn't seem strictly necessary for it to know the Rust unwind personality. It's perfectly fine (and even desirable) for it to just catch/suspend foreign unwinds the same way as Rust unwinds (that is, abort without cleaning up the unwind) rather than inspecting personality to determine if it's a foreign unwind (e.g. to instead print the "cannot catch foreign unwind" rtabort).

I forget the exact unwinding ABI(s) atm, though, so I don't recall if it's necessary for an unwind terminating frame is still required to specify a personality even if it wants to catch all unwinds including foreign.

(It'd be good to verify the behavior of #[rustc_nounwind] when hit by a C++ (or other foreign) unwind. Not to say it's necessarily broken, but just to ensure that it isn't.)

bjorn3 commented 1 year ago

Since #[rustc_nounwind] wants to catch and immediately abort on any unwind, it doesn't seem strictly necessary for it to know the Rust unwind personality.

The personality function is necessary to catch and abort. Any frame without a personality function is skipped entirely by the unwinder. The abort information is either in a landingpad, which execution is redirected to by the rust personality function or possibly in the future in the LSDA (language specific data area) which is interpreted by the personality function and is an opaque blob to the unwinder.

It's perfectly fine (and even desirable) for it to just catch/suspend foreign unwinds the same way as Rust unwinds (that is, abort without cleaning up the unwind)

For rust panics we never call into the unwinder in case of -Cpanic=abort. We literally swap out the function to initiate the rust panic with one which immediately aborts through the panic_abort crate. We can't replace the function to start unwinding for foreign exceptions.

CAD97 commented 1 year ago

So yes, it's not possible to catch/terminate an unwind without specifying a personality. I wasn't certain and thought a catch-all could be defined without, but apparently I misremembered. (It's also possible that's true in some unwind ABIs but not always in general e.g. as exposed by LLVM.)

aside clarification, off-topic to OP And to be clear, I wasn't meaning to say to change anything about how unwinds are started, just how `#[rustc_nounwind]` terminates unwinds (to prevent the function from unwinding). IIRC `extern "C-unwind"` immediately catches and aborts on a foreign unwind entering Rust when `-Cpanic=abort`[^1], so a foreign unwind hitting a `-Cpanic=abort`-mode `#[rustc_nounwind]` is an impossibility (or at least UB); the aside was off-topic for `-Cpanic=unwind`[^2] when foreign unwinds can reach it. [^2]: Catching foreign unwinds when `-Cpanic=unwind` is still of course necessary but easy to do when we've got access to the full unwinding runtime. [^1]: ... Since you say catching unwinds always requires a personality, I assume this means `extern "C-unwind"` still requires `_rust_eh_personality` even with `-Cpanic=abort -Zbuild-std`? That seems at least reasonable (you're explicitly asking to interface with unwinding) but still somewhat unfortunate, since no Rust unwinds will be triggered (on this side of FFI).
bjorn3 commented 1 year ago

I assume this means extern "C-unwind" still requires _rust_eh_personality even with -Cpanic=abort -Zbuild-std?

For C-unwind -> Rust, yes. For Rust -> C-unwind I believe we skip aborting as no panic/exception can reach it. Or at least we can do so in the future without too much effort I think.

RalfJung commented 1 year ago

Ah, that's unfortunate. :/ Sorry for the breakage.

We can also revert that PR for now. I won't have the time to do follow-on work here.

gngshn commented 1 year ago

It seems that there is no easy solution to this problem, Since #106045 does not introduce new features or fix any bugs, just behavioral optimization. Can we just revert it first as @RalfJung said, and discuss the consistency of no_std and std panic behavior later when @RalfJung has time to follow on it?

bjorn3 commented 1 year ago

Only if https://github.com/rust-lang/rust/pull/102318 is reverted too. Otherwise default_alloc_error_handler will get stabilized in a way that allows unwinding from the alloc error handler, which we wouldn't be able to revert back to aborting by landing https://github.com/rust-lang/rust/pull/106045 again.

gngshn commented 1 year ago

106045 only change the behavor in no_std environment. And it seems that almost all the no_std crates set panic=abort. Just only revert #106045 seems not bad?

bjorn3 commented 1 year ago

I guess

gngshn commented 1 year ago

If revert #106045 alone is not an acceptable option, then it is better to continue to wait for someone to fix this problem, and keep these two merged PRs at least allow bare metal development to use the alloc in rust 1.68 stable :)

RalfJung commented 1 year ago

later when @RalfJung has time to follow on it?

FWIW I don't think I would do the follow-on in the future either. This was meant to be a "quick fix". I hope someone else can take over. :)

bjorn3 commented 1 year ago

I'm trying one approach right now :)

Amanieu commented 1 year ago

The fundamental issue here is that the standard library (including core/alloc) is built with panic=unwind, and therefore has a dependency on the personality function. This personality function is never called with panic=abort, but the linker doesn't know that. This used to mostly work with LTO, but this was more by luck than by design.

One possible solution would be to have rustc automatically insert a dummy implementation of the personality function (with weak linkage) when building without the standard library (which provides the personality function even with panic=abort).

bjorn3 commented 1 year ago

This specific issue happens even when libcore is compiled with panic=abort due to a call to extern "Rust" { #[lang = "panic_impl"] fn panic_impl(&core::panic::PanicInfo<'_>); } from a #[rustc_nounwind] call requiring the personality function to abort if panic_impl does actually unwind. The solution to that I think is adding another panic_impl_nounwind lang item annotated with #[rustc_nounwind] itself such that the call doesn't need a personality function to abort. panic_impl_nounwind itself will only get a personality function if it calls something that can unwind, which for -Cpanic=abort is only an extern "C-unwind" or extern "Rust" extern function.

bjorn3 commented 1 year ago

One possible solution would be to have rustc automatically insert a dummy implementation of the personality function (with weak linkage) when building without the standard library (which provides the personality function even with panic=abort).

This can result in UB if extern "C-unwind" or extern "Rust" extern functions are called unless the personality function itself aborts, but the personality function on arm can be called when creating a backtrace, not just when unwinding. So on arm it shouldn't abort in case of state == _US_VIRTUAL_UNWIND_FRAME | _US_FORCE_UNWIND: https://github.com/rust-lang/rust/blob/38a76f33220c4b9d13dda1fa8f6c629c8a7bcc5d/library/std/src/personality/gcc.rs#L98-L101

bjorn3 commented 1 year ago

I've got a basic implementation in https://github.com/bjorn3/rust/tree/panic_handler_nounwind. Still has some failing tests though. However while working on this, I realized that we actually already correctly handle everything in the case of libcore compiled with -Cpanic=abort. In the case of libcore compiled with -Cpanic=unwind you should get

error: language item required, but not found: `eh_personality`
  |
  = note: this can occur when a binary crate with `#![no_std]` is compiled for a target where `eh_personality` is defined in the standard library
  = help: you may be able to compile for a target that doesn't need `eh_personality`, specify a target with `--target` or in `.cargo/config`

This is not what @gngshn got however despite not using -Zbuild-std to recompile libcore with -Cpanic=abort. Maybe this happened to work in the past due to all precompiled libcore functions being omitted as dead code? @gngshn could you provide a standalone reproducing example and say which cargo invocation you used to build it?

bjorn3 commented 1 year ago

For reference this is what worked for me:

# Cargo.toml
[package]
name = "foo"
version = "0.1.0"
edition = "2021"

[profile.dev]
panic = "abort"

[profile.release]
panic = "abort"
// src/main.rs
#![no_std]
#![no_main]

extern crate alloc;

#[link(name = "c")]
extern "C" {}

#[panic_handler]
fn panic_handler(_: &core::panic::PanicInfo<'_>) -> ! { loop {} }

struct A;
unsafe impl alloc::alloc::GlobalAlloc for A {
unsafe fn alloc(&self, _: alloc::alloc::Layout) -> *mut u8 { todo!() }
unsafe fn dealloc(&self, _: *mut u8, _: alloc::alloc::Layout) { todo!() }
}

#[global_allocator]
static ALLOC: A = A;

#[no_mangle]
extern "C" fn main() {
    alloc::alloc::handle_alloc_error(alloc::alloc::Layout::new::<()>());
}
$ cargo +nightly build -Zbuild-std=core,alloc --target x86_64-unknown-linux-gnu
   Compiling foo v0.1.0 (/tmp/foo)
    Finished release [optimized] target(s) in 0.16s
gngshn commented 1 year ago

Hello @bjorn3 I test three cases in my pc by using your example, I got:

$ cargo --version                          
cargo 1.66.1 (ad779e08b 2023-01-10)

$ cargo +nightly rustc -- --version
balaba.....
rustc 1.68.0-nightly (3984bc583 2023-01-17)

$ cargo +nightly build --release                                          
   Compiling github_issue_test v0.1.0 (/home/gngshn/develop/rust/github_issue_test)
error: linking with `cc` failed: exit status: 1
  |
  = note: balabalbala............
  = note: /home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-cb19371b39fc63d8.rlib(alloc-cb19371b39fc63d8.alloc.ea0e1c85-cgu.0.rcgu.o):(.data.DW.ref.rust_eh_personality[DW.ref.rust_eh_personality]+0x0): undefined reference to `rust_eh_personality'
          collect2: error: ld returned 1 exit status

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

$ cargo +nightly build -Zbuild-std=core,alloc --release --target x86_64-unknown-linux-gnu
   Compiling core v0.0.0 (/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core)
   Compiling compiler_builtins v0.1.85
   Compiling rustc-std-workspace-core v1.99.0 (/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/rustc-std-workspace-core)
   Compiling alloc v0.0.0 (/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc)
   Compiling github_issue_test v0.1.0 (/home/gngshn/develop/rust/github_issue_test)
    Finished release [optimized] target(s) in 9.77s

$ cargo +nightly-2022-12-29 build --release                       
   Compiling github_issue_test v0.1.0 (/home/gngshn/develop/rust/github_issue_test)
    Finished release [optimized] target(s) in 0.29s
bjorn3 commented 1 year ago

For me cargo +nightly build --release gave

error: language item required, but not found: `eh_personality`
  |
  = note: this can occur when a binary crate with `#![no_std]` is compiled for a target where `eh_personality` is defined in the standard library
  = help: you may be able to compile for a target that doesn't need `eh_personality`, specify a target with `--target` or in `.cargo/config`

and should have done so for years. Do you have any custom .cargo/config.toml changing flags or something?

gngshn commented 1 year ago

For me cargo +nightly build --release gave

error: language item required, but not found: `eh_personality`
  |
  = note: this can occur when a binary crate with `#![no_std]` is compiled for a target where `eh_personality` is defined in the standard library
  = help: you may be able to compile for a target that doesn't need `eh_personality`, specify a target with `--target` or in `.cargo/config`

and should have done so for years. Do you have any custom .cargo/config.toml changing flags or something?

No, my project is fresh generated by cargo new foo, and my ${HOME}/.cargo/config.toml content is below:

[source.crates-io]
replace-with = 'ustc'

[source.ustc]
registry = "git://mirrors.ustc.edu.cn/crates.io-index"

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-linux-gnu-gcc"

If I delete all content in ${HOME}/.cargo/config.toml, it is still show:

$ cargo +nightly build --release
   Compiling foo v0.1.0 (/home/gngshn/develop/rust/foo)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/gngshn/.vscode-server/bin/97dec172d3256f8ca4bfb2143f3f76b503ca0534/bin/remote-cli:/home/gngshn/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/gngshn/.oh-my-zsh/custom/plugins/fzf-zsh-plugin/bin:/home/gngshn/.fzf/bin:/home/gngshn/.local/bin:/home/gngshn/.cargo/bin:/home/gngshn/.local/opt/arm-none-eabi/bin:/home/gngshn/.local/opt/arm-none-linux-gnueabihf/bin:/home/gngshn/.local/opt/aarch64-none-linux-gnu/bin:/home/gngshn/.local/opt/riscv32-linux-gnu/bin:/home/gngshn/.oh-my-zsh/custom/plugins/fzf-zsh-plugin/bin:/home/gngshn/.local/bin:/home/gngshn/.cargo/bin:/home/gngshn/.local/opt/arm-none-eabi/bin:/home/gngshn/.local/opt/arm-none-linux-gnueabihf/bin:/home/gngshn/.local/opt/aarch64-none-linux-gnu/bin:/home/gngshn/.local/opt/riscv32-linux-gnu/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcgGFeS6/symbols.o" "/home/gngshn/develop/rust/foo/target/release/deps/foo-f850b37a340e0123.foo.5b0169bd-cgu.0.rcgu.o" "/home/gngshn/develop/rust/foo/target/release/deps/foo-f850b37a340e0123.foo.5b0169bd-cgu.1.rcgu.o" "/home/gngshn/develop/rust/foo/target/release/deps/foo-f850b37a340e0123.21fmwiugw2abqahh.rcgu.o" "-Wl,--as-needed" "-L" "/home/gngshn/develop/rust/foo/target/release/deps" "-L" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-lc" "-Wl,-Bstatic" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-cb19371b39fc63d8.rlib" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-522518611024dce5.rlib" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-05898138a596088a.rlib" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-66b9c3ae5ff29c13.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/gngshn/develop/rust/foo/target/release/deps/foo-f850b37a340e0123" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs"
  = note: /usr/bin/ld: /home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-cb19371b39fc63d8.rlib(alloc-cb19371b39fc63d8.alloc.b0be2ffa-cgu.0.rcgu.o):(.data.DW.ref.rust_eh_personality[DW.ref.rust_eh_personality]+0x0): undefined reference to `rust_eh_personality'
          collect2: error: ld returned 1 exit status

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: could not compile `foo` due to previous error

Before I test this, I have executed rustup toolchain update which updated nightly toolchain to 2023-01-18.

gngshn commented 1 year ago

By the way, my system is ubuntu 22.04 WSL. I also tested this case on MacOS with mbp and ubuntu 18.04 with PC, it got almost the same result.

$ cat /etc/lsb-release                     
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
bjorn3 commented 1 year ago

I'm utterly confused how this didn't produce that error: language item required, but not found: `eh_personality` error for you.

gngshn commented 1 year ago

I can get error: language item required, but not found: `eh_personality` only if I remove panic = "abort" in my cargo.toml.

Maybe you can compare your verbose log with mine to find some difference?

$ cargo +nightly -vv build --release
   Compiling foo v0.1.0 (/home/gngshn/develop/rust/foo)
     Running `CARGO=/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo CARGO_BIN_NAME=foo CARGO_CRATE_NAME=foo CARGO_MANIFEST_DIR=/home/gngshn/develop/rust/foo CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=foo CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 LD_LIBRARY_PATH='/home/gngshn/develop/rust/foo/target/release/deps:/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib' rustc --crate-name foo --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=266 --crate-type bin --emit=dep-info,link -C opt-level=3 -C panic=abort -C embed-bitcode=no -C metadata=f850b37a340e0123 -C extra-filename=-f850b37a340e0123 --out-dir /home/gngshn/develop/rust/foo/target/release/deps -L dependency=/home/gngshn/develop/rust/foo/target/release/deps`
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/home/gngshn/.vscode-server/bin/97dec172d3256f8ca4bfb2143f3f76b503ca0534/bin/remote-cli:/home/gngshn/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/gngshn/.oh-my-zsh/custom/plugins/fzf-zsh-plugin/bin:/home/gngshn/.fzf/bin:/home/gngshn/.local/bin:/home/gngshn/.cargo/bin:/home/gngshn/.local/opt/arm-none-eabi/bin:/home/gngshn/.local/opt/arm-none-linux-gnueabihf/bin:/home/gngshn/.local/opt/aarch64-none-linux-gnu/bin:/home/gngshn/.local/opt/riscv32-linux-gnu/bin:/home/gngshn/.oh-my-zsh/custom/plugins/fzf-zsh-plugin/bin:/home/gngshn/.local/bin:/home/gngshn/.cargo/bin:/home/gngshn/.local/opt/arm-none-eabi/bin:/home/gngshn/.local/opt/arm-none-linux-gnueabihf/bin:/home/gngshn/.local/opt/aarch64-none-linux-gnu/bin:/home/gngshn/.local/opt/riscv32-linux-gnu/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcUr8Amy/symbols.o" "/home/gngshn/develop/rust/foo/target/release/deps/foo-f850b37a340e0123.foo.5b0169bd-cgu.0.rcgu.o" "/home/gngshn/develop/rust/foo/target/release/deps/foo-f850b37a340e0123.foo.5b0169bd-cgu.1.rcgu.o" "/home/gngshn/develop/rust/foo/target/release/deps/foo-f850b37a340e0123.21fmwiugw2abqahh.rcgu.o" "-Wl,--as-needed" "-L" "/home/gngshn/develop/rust/foo/target/release/deps" "-L" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-lc" "-Wl,-Bstatic" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-cb19371b39fc63d8.rlib" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-522518611024dce5.rlib" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-05898138a596088a.rlib" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-66b9c3ae5ff29c13.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/gngshn/develop/rust/foo/target/release/deps/foo-f850b37a340e0123" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs"
  = note: /usr/bin/ld: /home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-cb19371b39fc63d8.rlib(alloc-cb19371b39fc63d8.alloc.b0be2ffa-cgu.0.rcgu.o):(.data.DW.ref.rust_eh_personality[DW.ref.rust_eh_personality]+0x0): undefined reference to `rust_eh_personality'
          collect2: error: ld returned 1 exit status

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: could not compile `foo` due to previous error

Caused by:
  process didn't exit successfully: `CARGO=/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo CARGO_BIN_NAME=foo CARGO_CRATE_NAME=foo CARGO_MANIFEST_DIR=/home/gngshn/develop/rust/foo CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=foo CARGO_PKG_REPOSITORY='' CARGO_PKG_RUST_VERSION='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 LD_LIBRARY_PATH='/home/gngshn/develop/rust/foo/target/release/deps:/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib:/home/gngshn/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib' rustc --crate-name foo --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=266 --crate-type bin --emit=dep-info,link -C opt-level=3 -C panic=abort -C embed-bitcode=no -C metadata=f850b37a340e0123 -C extra-filename=-f850b37a340e0123 --out-dir /home/gngshn/develop/rust/foo/target/release/deps -L dependency=/home/gngshn/develop/rust/foo/target/release/deps` (exit status: 1)
bjorn3 commented 1 year ago

I can now reproduce it too for some reason. I can also reproduce that it worked on stable (if I add #![feature(default_alloc_error_handler)] and use the bootstrap trick) However even on stable it is really fragile. Adding alloc::format!("{:?}", "a"); to main.rs before handle_alloc_error is enough to make it fail with a linker error again. Except now complaining about both rust_eh_personality and _Unwind_Resume. Adding -Clink-dead-code is another way to make it fail. In other words you just got lucky that it happened to work. It just so happened that all standard library functions needing a personality function due to being compiled with -Cpanic=unwind happened to get garbage collected during linking.

I think @Amanieu's suggestion of creating a dummy personality function would be the proper solution. As I said in https://github.com/rust-lang/rust/issues/106864#issuecomment-1385813361 it will need to abort in all cases except when generating backtraces.

gngshn commented 1 year ago

I can now reproduce it too for some reason. I can also reproduce that it worked on stable (if I add #![feature(default_alloc_error_handler)] and use the bootstrap trick) However even on stable it is really fragile. Adding alloc::format!("{:?}", "a"); to main.rs before handle_alloc_error is enough to make it fail with a linker error again. Except now complaining about both rust_eh_personality and _Unwind_Resume. Adding -Clink-dead-code is another way to make it fail. In other words you just got lucky that it happened to work. It just so happened that all standard library functions needing a personality function due to being compiled with -Cpanic=unwind happened to get garbage collected during linking.

I think @Amanieu's suggestion of creating a dummy personality function would be the proper solution. As I said in #106864 (comment) it will need to abort in all cases except when generating backtraces.

Great, this is the magic of linker gc and my good luck~

I naively thought this would always compile properly :(

nbdd0121 commented 1 year ago

Given Ralf's PR and the default_alloc_error_handler stabilisation PR will be in the same release, technically we are not breaking non-nightly code? Especially as @bjorn3 said it used to work by luck.

Do we need to revert Ralf's PR for beta, so would it be fine to leave this as is?

gngshn commented 1 year ago

I don't think we need revert Ralf's PR now! Looking forward @bjorn3's great work to solve the current compilation problem.

bjorn3 commented 1 year ago

The solution for which I posted a branch in https://github.com/rust-lang/rust/issues/106864#issuecomment-1387433545 is not something I will finish I think given what I now know about what the actual issue is. I may implement the solution in https://github.com/rust-lang/rust/issues/106864#issuecomment-1385796971 at a later point, but it is probably a bit harder, so I don't think I have time for it right now. I have added it to my backlog, but it might take a while until I get to it. If someone else would like to pick it up that did be great.

gngshn commented 1 year ago

Hi @bjorn3
Thank you for your support! Although it's still not possible to use stable no_std rust in an OS environment, we can do some tests with build-std in nightly rust and cargo!

Noratrieb commented 11 months ago

I'm utterly confused how this didn't produce that error: language item required, but not found: `eh_personality` error for you.

https://github.com/rust-lang/rust/blob/c3efa51947c3b14652d1e85dd3bb406522c8af22/compiler/rustc_middle/src/middle/lang_items.rs#L48-L58

apparently this logic thinks that eh_personality is not necessary when using panic=abort.

bjorn3 commented 11 months ago

If libcore is compiled as panic=abort and extern "C-unwind" isn't used anywhere then eh_personality is indeed not necessary. If any crate is compiled as panic=unwind andextern "C-unwind"` isn't used, then eh_personality isn't reachable, so having rustc produce a dummy impl should be fine.

2moe commented 10 months ago

I have this issue too. The easiest way is to modify the Cargo.toml. 

[profile.release]
lto = "fat"
panic = "abort"

Note: lto = "thin" still has this issue. BTW, rustc & cargo version:

rustc: 1.74.1 (a28077b28 2023-12-04)
cargo: 1.74.1 (ecb9851af 2023-10-18)
host: x86_64-unknown-linux-gnu

However, during development, I can not always enable Fat LTO + release for compilation speed. So I've tried to work on some kinda "hacky" methods.

#![allow(internal_features)]
#![cfg_attr(debug_assertions, feature(lang_items))]

#[cfg(debug_assertions)]
#[lang = "eh_personality"]
extern "C" fn rust_eh_personality() {}

It requires nightly toolchain.

As with thin LTO, some codes work and some don't.

It works:

use ::alloc::boxed::Box;
let b = Box::new("Heap");
println!("{b}");

It does not work:

use libc_print::std_name::println;

let up = "Hello world".to_uppercase();
println!("{up}");

To avoid using the nightly toolchain, and to solve the linking problem. I found that simply creating the relevant function manually solved the problem.

#[no_mangle]
extern "C" fn rust_eh_personality() {}

#[allow(non_snake_case)]
#[no_mangle]
extern "C" fn _Unwind_Resume() {}

After this, even stable toolchain + dev profile (debug mode) will compile successfully.

The complete demo code is below:

#![no_std]
#![no_main]
//!
//! ```cargo
//! [dependencies]
//! libc = { version = "0.2.151", default-features = false }
//! libc-print = "0.1.22"
//! libc_alloc = "1.0.5"
//!
//! [profile.dev]
//! panic = "abort"
//!
//! [profile.release]
//! # lto = "thin"
//! panic = "abort"
//! opt-level = "s"
//! strip = true
//! ```
//!
#[no_mangle]
extern "C" fn rust_eh_personality() {}

#[allow(non_snake_case)]
#[no_mangle]
extern "C" fn _Unwind_Resume() {}
// ---------

extern crate alloc;
use ::alloc::boxed::Box;
use ::core::{ffi::c_int, panic::PanicInfo};
use libc_alloc::LibcAlloc;
use libc_print::std_name::{dbg, eprintln, println};

#[global_allocator]
static GLOBAL_ALLOC: LibcAlloc = LibcAlloc;

#[panic_handler]
fn panic(info: &PanicInfo) -> ! {
    eprintln!("[FATAL]: {info}");
    unsafe { libc::abort() }
}

#[no_mangle]
pub extern "C" fn main() -> c_int {
    let upper = "ok".to_uppercase();
    println!("{upper}");

    let b = Box::new("heap");
    dbg!(b);
    0
}

I don't know if it would be a problem to manually create a #[no_mangle] fn _Unwind_Resume(), because it looks really weird.

gngshn commented 10 months ago

I have this issue too. The easiest way is to modify the Cargo.toml.

[profile.release]
lto = "fat"
panic = "abort"

Note: lto = "thin" still has this issue. BTW, rustc & cargo version:

rustc: 1.74.1 (a28077b28 2023-12-04)
cargo: 1.74.1 (ecb9851af 2023-10-18)
host: x86_64-unknown-linux-gnu

However, during development, I can not always enable Fat LTO + release for compilation speed. So I've tried to work on some kinda "hacky" methods.

#![allow(internal_features)]
#![cfg_attr(debug_assertions, feature(lang_items))]

#[cfg(debug_assertions)]
#[lang = "eh_personality"]
extern "C" fn rust_eh_personality() {}

It requires nightly toolchain.

As with thin LTO, some codes work and some don't.

It works:

use ::alloc::boxed::Box;
let b = Box::new("Heap");
println!("{b}");

It does not work:

use libc_print::std_name::println;

let up = "Hello world".to_uppercase();
println!("{up}");

To avoid using the nightly toolchain, and to solve the linking problem. I found that simply creating the relevant function manually solved the problem.

#[no_mangle]
extern "C" fn rust_eh_personality() {}

#[allow(non_snake_case)]
#[no_mangle]
extern "C" fn _Unwind_Resume() {}

After this, even stable toolchain + dev profile (debug mode) will compile successfully.

The complete demo code is below:

#![no_std]
#![no_main]
//!
//! ```cargo
//! [dependencies]
//! libc = { version = "0.2.151", default-features = false }
//! libc-print = "0.1.22"
//! libc_alloc = "1.0.5"
//!
//! [profile.dev]
//! panic = "abort"
//!
//! [profile.release]
//! # lto = "thin"
//! panic = "abort"
//! opt-level = "s"
//! strip = true
//! ```
//!
#[no_mangle]
extern "C" fn rust_eh_personality() {}

#[allow(non_snake_case)]
#[no_mangle]
extern "C" fn _Unwind_Resume() {}
// ---------

extern crate alloc;
use ::alloc::boxed::Box;
use ::core::{ffi::c_int, panic::PanicInfo};
use libc_alloc::LibcAlloc;
use libc_print::std_name::{dbg, eprintln, println};

#[global_allocator]
static GLOBAL_ALLOC: LibcAlloc = LibcAlloc;

#[panic_handler]
fn panic(info: &PanicInfo) -> ! {
    eprintln!("[FATAL]: {info}");
    unsafe { libc::abort() }
}

#[no_mangle]
pub extern "C" fn main() -> c_int {
    let upper = "ok".to_uppercase();
    println!("{upper}");

    let b = Box::new("heap");
    dbg!(b);
    0
}

I don't know if it would be a problem to manually create a #[no_mangle] fn _Unwind_Resume(), because it looks really weird.

Thanks for your infomation. This seems another good way to test no_std + liballoc on stable toolchain.

As you said, doing this seems to replace the default implementation of _Unwind_Resume and rust_eh_personality in rust static lib. This also can not be used in production environment. It maybe UB? And the naming _Unwind_Resume may be changed in the future?

We also can use cargo build std with RUSTC_BOOTSTRAP=1, But I think this is also not a real safty way and can not be used in production environment.

bjorn3 commented 9 months ago

And the naming _Unwind_Resume may be changed in the future?

Nope, that is a stable interface of the itanium style unwinding that is used on almost all Unix systems.

It maybe UB?

If your program is linking in C++ code that wants to throw exceptions, yes. But in that case you would link against libunwind anyway and thus not have this undefined reference.

As for rust_eh_personality, that one is indeed unstable though.

By the way all targets that are meant exclusively for no_std usage don't have this issue as the standard library is already compiled with panic=abort. It is a real problem for the targets that are for running under a conventional OS though.

GrigorenkoPV commented 7 months ago

Might be related to #56152