LTO coordinator panics by unwrapping Option::None

[knoellle]

Code

Discovered this error while fuzzing for https://github.com/rust-lang/rust/issues/81280

I used cargo watch -- cargo build --profile incremental for building this binary crate, while touch-ing random files at random intervals to trigger rebuilds.

Fish script:

while true
    touch (find src | shuf -n1)
    sleep 0.(random 1 999)
end

Meta

rustc --version --verbose:

rustc 1.81.0 (eeb90cda1 2024-09-04)
binary: rustc
commit-hash: eeb90cda1969383f56a2637cbd3037bdf598841c
commit-date: 2024-09-04
host: x86_64-unknown-linux-gnu
release: 1.81.0
LLVM version: 18.1.7

Error output

thread 'coordinator' panicked at compiler/rustc_codegen_llvm/src/back/lto.rs:836:36:
called `Option::unwrap()` on a `None` value

Backtrace

``` called `Option::unwrap()` on a `None` value stack backtrace: 0: 0x7f5bbd68a3e5 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5 1: 0x7f5bbd68a3e5 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 2: 0x7f5bbd68a3e5 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:65:5 3: 0x7f5bbd68a3e5 - ::fmt::h1b9dad2a88e955ff at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:40:26 4: 0x7f5bbd6d9eeb - core::fmt::rt::Argument::fmt::h351a7824f737a6a0 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/rt.rs:173:76 5: 0x7f5bbd6d9eeb - core::fmt::write::h4b5a1270214bc4a7 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/mod.rs:1182:21 6: 0x7f5bbd67ef6f - std::io::Write::write_fmt::hd04af345a50c312d at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/io/mod.rs:1827:15 7: 0x7f5bbd68cbd1 - std::sys::backtrace::BacktraceLock::print::h68d41b51481bce5c at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:43:9 8: 0x7f5bbd68cbd1 - std::panicking::default_hook::{{closure}}::h96ab15e9936be7ed at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:269:22 9: 0x7f5bbd68c8ac - std::panicking::default_hook::h3cacb9c27561ad33 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:296:9 10: 0x7f5bb9c93420 - std[1f2242ed6435445e]::panicking::update_hook::>::{closure#0} 11: 0x7f5bbd68d59f - as core::ops::function::Fn>::call::hce7569f4ca5d1b64 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2084:9 12: 0x7f5bbd68d59f - std::panicking::rust_panic_with_hook::hfe205f6954b2c97b at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:808:13 13: 0x7f5bbd68d193 - std::panicking::begin_panic_handler::{{closure}}::h6cb44b3a50f28c44 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:667:13 14: 0x7f5bbd68a8a9 - std::sys::backtrace::__rust_end_short_backtrace::hf1c1f2a92799bb0e at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:168:18 15: 0x7f5bbd68ce54 - rust_begin_unwind at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:665:5 16: 0x7f5bbd6d64a3 - core::panicking::panic_fmt::h3d8fc78294164da7 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:74:14 17: 0x7f5bbd6d652c - core::panicking::panic::hec978767ec2d35ff at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/panicking.rs:148:5 18: 0x7f5bbd6d6299 - core::option::unwrap_failed::hba6b08832f9ce30b at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/option.rs:2020:5 19: 0x7f5bbbf72a6b - rustc_codegen_llvm[5ff2375b9bf6d639]::back::lto::thin_lto 20: 0x7f5bbbf6d00b - ::run_thin_lto 21: 0x7f5bbc1113ce - rustc_codegen_ssa[382a85cbfbe8c2f6]::back::write::start_executing_work::::{closure#5} 22: 0x7f5bbc10f876 - std[1f2242ed6435445e]::sys::backtrace::__rust_begin_short_backtrace::<::spawn_named_thread::{closure#5}, core[3cad2706d8bdcdc4]::result::Result>::{closure#0}, core[3cad2706d8bdcdc4]::result::Result> 23: 0x7f5bbc10f6ad - <::spawn_unchecked_<::spawn_named_thread::{closure#5}, core[3cad2706d8bdcdc4]::result::Result>::{closure#0}, core[3cad2706d8bdcdc4]::result::Result>::{closure#1} as core[3cad2706d8bdcdc4]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} 24: 0x7f5bbd6975fb - as core::ops::function::FnOnce>::call_once::ha1963004222e7822 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2070:9 25: 0x7f5bbd6975fb - as core::ops::function::FnOnce>::call_once::h1086ced1f7c494c2 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2070:9 26: 0x7f5bbd6975fb - std::sys::pal::unix::thread::Thread::new::thread_start::ha8af9c992ef0b208 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/pal/unix/thread.rs:108:17 27: 0x7f5bbd466897 - start_thread 28: 0x7f5bbd4eda5c - __GI___clone3 29: 0x0 - error: the compiler unexpectedly panicked. this is a bug. note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md note: rustc 1.81.0 (eeb90cda1 2024-09-04) running on x86_64-unknown-linux-gnu note: compiler flags: --crate-type bin -C opt-level=3 -C embed-bitcode=no -C incremental=[REDACTED] -C strip=debuginfo note: some of the compiler flags provided by cargo are hidden query stack during panic: end of query stack thread 'rustc' panicked at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/compiler/rustc_codegen_ssa/src/back/write.rs:2057:17: /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/compiler/rustc_codegen_ssa/src/back/write.rs:2057:17: panic during codegen/LLVM phase stack backtrace: 0: 0x7f5bbd68a3e5 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5 1: 0x7f5bbd68a3e5 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 2: 0x7f5bbd68a3e5 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:65:5 3: 0x7f5bbd68a3e5 - ::fmt::h1b9dad2a88e955ff at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:40:26 4: 0x7f5bbd6d9eeb - core::fmt::rt::Argument::fmt::h351a7824f737a6a0 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/rt.rs:173:76 5: 0x7f5bbd6d9eeb - core::fmt::write::h4b5a1270214bc4a7 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/fmt/mod.rs:1182:21 6: 0x7f5bbd67ef6f - std::io::Write::write_fmt::hd04af345a50c312d at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/io/mod.rs:1827:15 7: 0x7f5bbd68cbd1 - std::sys::backtrace::BacktraceLock::print::h68d41b51481bce5c at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/sys/backtrace.rs:43:9 8: 0x7f5bbd68cbd1 - std::panicking::default_hook::{{closure}}::h96ab15e9936be7ed at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:269:22 9: 0x7f5bbd68c8ac - std::panicking::default_hook::h3cacb9c27561ad33 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:296:9 10: 0x7f5bb9c93420 - std[1f2242ed6435445e]::panicking::update_hook::>::{closure#0} 11: 0x7f5bbd68d59f - as core::ops::function::Fn>::call::hce7569f4ca5d1b64 at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/alloc/src/boxed.rs:2084:9 12: 0x7f5bbd68d59f - std::panicking::rust_panic_with_hook::hfe205f6954b2c97b at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/panicking.rs:808:13 13: 0x7f5bba236ae2 - std[1f2242ed6435445e]::panicking::begin_panic::::{closure#0} 14: 0x7f5bba22f626 - std[1f2242ed6435445e]::sys::backtrace::__rust_end_short_backtrace::::{closure#0}, !> 15: 0x7f5bba221201 - std[1f2242ed6435445e]::panicking::begin_panic:: 16: 0x7f5bba2b98cd - rustc_middle[ba2289ab3ae064d4]::util::bug::opt_span_bug_fmt::::{closure#0} 17: 0x7f5bba29f49a - rustc_middle[ba2289ab3ae064d4]::ty::context::tls::with_opt::::{closure#0}, !>::{closure#0} 18: 0x7f5bba29f31b - rustc_middle[ba2289ab3ae064d4]::ty::context::tls::with_context_opt::::{closure#0}, !>::{closure#0}, !> 19: 0x7f5bb7c3a600 - rustc_middle[ba2289ab3ae064d4]::util::bug::bug_fmt 20: 0x7f5bbc1081d0 - ::join_codegen 21: 0x7f5bbc1030c3 - ::link 22: 0x7f5bbbf2d55c - rustc_interface[53a414ae04dc6ffb]::interface::run_compiler::, rustc_driver_impl[8683aa37472b7dde]::run_compiler::{closure#0}>::{closure#1} 23: 0x7f5bbbe7395b - std[1f2242ed6435445e]::sys::backtrace::__rust_begin_short_backtrace::, rustc_driver_impl[8683aa37472b7dde]::run_compiler::{closure#0}>::{closure#1}, core[3cad2706d8bdcdc4]::result::Result<(), rustc_span[28a649581f99a5bd]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[3cad2706d8bdcd ```

[saethlin]

while touch-ing random files at random intervals to trigger rebuilds.

FYI this will cause almost none of the incremental compilation code in the compiler to do anything interesting; random edits that are then reverted would tickle a lot more. It would also be slower.

It looks to me like you somehow ended up with an LLVM module (codegen unit) whose name is not UTF-8. That is wild. https://github.com/rust-lang/rust/pull/130680 will improve the ICE message. If you're willing to apply that patch locally, I'd love to see what the module name you're coming up with is.

[knoellle]

Thanks for the hint regarding my methodology.

Im building the 1.81 release with your commit cherry-picked on top now according to these instruction. I have never built rust from source before, Am I correct in assuming that I can run x.py install with the prefix path set to an arbitrary directory which I then use in myl PATH before running my build command again?

Or will that detect that its not the same compiler anymore and try to build everything anew, losing the broken state? If so, what would I need to do to pretend its still the same compiler version?

[saethlin]

I don't know anything about x install, I've never used it. I only know basically the quickstart steps: https://rustc-dev-guide.rust-lang.org/building/quickstart.html x build with rustup toolchain link lets you do cargo +stage1 build.

I'm also not sure I'd bother trying to make compiler not rebuild its artifacts. If you're fuzzing, eventually you should run into the same bad state again. Or you can show me how to run your fuzzer, and I'll run it overnight or on a cloud VM for a while.

[knoellle]

As suspected, the compiler didn't like the state and rebuilt everything. The reported error no longer occurs, but I'm trying to reproduce it now.

All of my fuzzing "tooling" and binary crate I was building (with commit hash) is in the issue description above if you want to run it yourself. Basically just run the cargo watch in one shell and the while loop in another (fish) shell.

Since I'd guess its a timing issue, i.e. the compilation process being interrupted at some critical stage that isn't protected well enough, I'm not sure how well this will be reproducible on my machine (8845HS CPU) or even in a cloud VM. I'll also let this run over night, so we'll see. I did make a backup of the tainted target directory that produces the issue with the 1.81 release though, just in case.