ICE: "no saved object file in work product" - potential incremental issue

[jamesmunns]

Code

This commit: https://github.com/memorysafety/river/commit/5ff346dd17524dff03a560517312caa7840569b2

Meta

This didn't seem to reproduce after a cargo clean, so I can't provide too much more wrt what caused this to happen.

So far I've only seen this once.

This was running in an aarch64 fedora VM running on an M2 Mac inside of UTM.

Error output

[Running 'cargo check && cargo test']
    Checking river v0.2.0 (/mnt/share/vmshare/contracts/isrg/river/source/river)
    Finished dev [unoptimized + debuginfo] target(s) in 0.63s
   Compiling river v0.2.0 (/mnt/share/vmshare/contracts/isrg/river/source/river)
thread 'cpy 1ukymu2hvb0r0otf' panicked at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/compiler/rustc_codegen_ssa/src/back/write.rs:917:44:
no saved object file in work product
stack backtrace:
   0:     0xffffab7439ec - std::backtrace_rs::backtrace::libunwind::trace::ha909211bcabbe3ac
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0xffffab7439ec - std::backtrace_rs::backtrace::trace_unsynchronized::h8f4bf76df48e0fca
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0xffffab7439ec - std::sys_common::backtrace::_print_fmt::h7a880f791e80e899
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:68:5
   3:     0xffffab7439ec - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h435dc28c83034320
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:44:22
   4:     0xffffab78d1e8 - core::fmt::rt::Argument::fmt::h22144fbf1e72c1e0
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/fmt/rt.rs:142:9
   5:     0xffffab78d1e8 - core::fmt::write::hd8738e3694a112d7
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/fmt/mod.rs:1120:17
   6:     0xffffab73944c - std::io::Write::write_fmt::h436b94c6697db76d
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/io/mod.rs:1846:15
   7:     0xffffab743818 - std::sys_common::backtrace::_print::ha54e28ed84d016bd
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:47:5
   8:     0xffffab743818 - std::sys_common::backtrace::print::h53c5489f7bece51d
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:34:9
   9:     0xffffab746284 - std::panicking::default_hook::{{closure}}::h0cf4d4378fa545a3
  10:     0xffffab745fb8 - std::panicking::default_hook::h083086c8c3d6f7ab
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/panicking.rs:292:9
  11:     0xffffac2973f8 - <alloc[f06ece74a4fc239f]::boxed::Box<rustc_driver_impl[acc3693e9fb6d00]::install_ice_hook::{closure#0}> as core[78d0b6180d7076d4]::ops::function::Fn<(&dyn for<'a, 'b> core[78d0b6180d7076d4]::ops::function::Fn<(&'a core[78d0b6180d7076d4]::panic::panic_info::PanicInfo<'b>,), Output = ()> + core[78d0b6180d7076d4]::marker::Sync + core[78d0b6180d7076d4]::marker::Send, &core[78d0b6180d7076d4]::panic::panic_info::PanicInfo)>>::call
  12:     0xffffab74696c - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hf2edefd0c36edede
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/alloc/src/boxed.rs:2029:9
  13:     0xffffab74696c - std::panicking::rust_panic_with_hook::h7eda7902743f1889
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/panicking.rs:785:13
  14:     0xffffab7466e8 - std::panicking::begin_panic_handler::{{closure}}::h9e0c8c22f2412703
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/panicking.rs:659:13
  15:     0xffffab743ebc - std::sys_common::backtrace::__rust_end_short_backtrace::ha7fb929dac979860
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:171:18
  16:     0xffffab746488 - rust_begin_unwind
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/panicking.rs:647:5
  17:     0xffffab7114bc - core::panicking::panic_fmt::hbb22b1130d3a7f38
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/panicking.rs:72:14
  18:     0xffffab71148c - core::panicking::panic_display::hc01b522008d00daf
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/panicking.rs:196:5
  19:     0xffffab71148c - core::panicking::panic_str::h887dc2e6a7ca69c2
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/panicking.rs:171:5
  20:     0xffffab71148c - core::option::expect_failed::h5639e0be8ab85f68
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/option.rs:1988:5
  21:     0xffffac5042fc - rustc_codegen_ssa[79a1e11a865bc274]::back::write::execute_copy_from_cache_work_item::<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend>
  22:     0xffffac5ad004 - std[67261b68e235c13b]::sys_common::backtrace::__rust_begin_short_backtrace::<<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend as rustc_codegen_ssa[79a1e11a865bc274]::traits::backend::ExtraBackendMethods>::spawn_named_thread<rustc_codegen_ssa[79a1e11a865bc274]::back::write::spawn_work<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend>::{closure#0}, ()>::{closure#0}, ()>
  23:     0xffffac505fec - <<std[67261b68e235c13b]::thread::Builder>::spawn_unchecked_<<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend as rustc_codegen_ssa[79a1e11a865bc274]::traits::backend::ExtraBackendMethods>::spawn_named_thread<rustc_codegen_ssa[79a1e11a865bc274]::back::write::spawn_work<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend>::{closure#0}, ()>::{closure#0}, ()>::{closure#1} as core[78d0b6180d7076d4]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  24:     0xffffab74f3b4 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h99a8d905b3363c45
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/alloc/src/boxed.rs:2015:9
  25:     0xffffab74f3b4 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hdc482d37839e1180
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/alloc/src/boxed.rs:2015:9
  26:     0xffffab74f3b4 - std::sys::pal::unix::thread::Thread::new::thread_start::he6d6f33e3183ae4d
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys/pal/unix/thread.rs:108:17
  27:     0xffffab590584 - start_thread
  28:     0xffffab5ffd4c - thread_start
  29:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.77.0 (aedd173a2 2024-03-17) running on aarch64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
thread 'coordinator' panicked at compiler/rustc_middle/src/util/bug.rs:36:26:
/rustc/aedd173a2c086e558c2b66d3743b344f977621a7/compiler/rustc_codegen_ssa/src/back/write.rs:1588:29: worker thread panicked
stack backtrace:
   0:     0xffffab7439ec - std::backtrace_rs::backtrace::libunwind::trace::ha909211bcabbe3ac
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0xffffab7439ec - std::backtrace_rs::backtrace::trace_unsynchronized::h8f4bf76df48e0fca
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0xffffab7439ec - std::sys_common::backtrace::_print_fmt::h7a880f791e80e899
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:68:5
   3:     0xffffab7439ec - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h435dc28c83034320
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:44:22
   4:     0xffffab78d1e8 - core::fmt::rt::Argument::fmt::h22144fbf1e72c1e0
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/fmt/rt.rs:142:9
   5:     0xffffab78d1e8 - core::fmt::write::hd8738e3694a112d7
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/fmt/mod.rs:1120:17
   6:     0xffffab73944c - std::io::Write::write_fmt::h436b94c6697db76d
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/io/mod.rs:1846:15
   7:     0xffffab743818 - std::sys_common::backtrace::_print::ha54e28ed84d016bd
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:47:5
   8:     0xffffab743818 - std::sys_common::backtrace::print::h53c5489f7bece51d
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:34:9
   9:     0xffffab746284 - std::panicking::default_hook::{{closure}}::h0cf4d4378fa545a3
  10:     0xffffab745fb8 - std::panicking::default_hook::h083086c8c3d6f7ab
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/panicking.rs:292:9
  11:     0xffffac2973f8 - <alloc[f06ece74a4fc239f]::boxed::Box<rustc_driver_impl[acc3693e9fb6d00]::install_ice_hook::{closure#0}> as core[78d0b6180d7076d4]::ops::function::Fn<(&dyn for<'a, 'b> core[78d0b6180d7076d4]::ops::function::Fn<(&'a core[78d0b6180d7076d4]::panic::panic_info::PanicInfo<'b>,), Output = ()> + core[78d0b6180d7076d4]::marker::Sync + core[78d0b6180d7076d4]::marker::Send, &core[78d0b6180d7076d4]::panic::panic_info::PanicInfo)>>::call
  12:     0xffffab74696c - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hf2edefd0c36edede
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/alloc/src/boxed.rs:2029:9
  13:     0xffffab74696c - std::panicking::rust_panic_with_hook::h7eda7902743f1889
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/panicking.rs:785:13
  14:     0xffffb0f945c0 - std[67261b68e235c13b]::panicking::begin_panic::<alloc[f06ece74a4fc239f]::string::String>::{closure#0}
  15:     0xffffb0f94274 - std[67261b68e235c13b]::sys_common::backtrace::__rust_end_short_backtrace::<std[67261b68e235c13b]::panicking::begin_panic<alloc[f06ece74a4fc239f]::string::String>::{closure#0}, !>
  16:     0xffffac1a7bcc - std[67261b68e235c13b]::panicking::begin_panic::<alloc[f06ece74a4fc239f]::string::String>
  17:     0xffffb1004b8c - rustc_middle[a0df8d032def7fe]::util::bug::opt_span_bug_fmt::<rustc_span[a3ab065b480aa793]::span_encoding::Span>::{closure#0}
  18:     0xffffb1002bf4 - rustc_middle[a0df8d032def7fe]::ty::context::tls::with_opt::<rustc_middle[a0df8d032def7fe]::util::bug::opt_span_bug_fmt<rustc_span[a3ab065b480aa793]::span_encoding::Span>::{closure#0}, !>::{closure#0}
  19:     0xffffb1002bc4 - rustc_middle[a0df8d032def7fe]::ty::context::tls::with_context_opt::<rustc_middle[a0df8d032def7fe]::ty::context::tls::with_opt<rustc_middle[a0df8d032def7fe]::util::bug::opt_span_bug_fmt<rustc_span[a3ab065b480aa793]::span_encoding::Span>::{closure#0}, !>::{closure#0}, !>
  20:     0xffffac1a02f8 - rustc_middle[a0df8d032def7fe]::util::bug::bug_fmt
  21:     0xffffac5aeec4 - std[67261b68e235c13b]::sys_common::backtrace::__rust_begin_short_backtrace::<<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend as rustc_codegen_ssa[79a1e11a865bc274]::traits::backend::ExtraBackendMethods>::spawn_named_thread<rustc_codegen_ssa[79a1e11a865bc274]::back::write::start_executing_work<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend>::{closure#5}, core[78d0b6180d7076d4]::result::Result<rustc_codegen_ssa[79a1e11a865bc274]::back::write::CompiledModules, ()>>::{closure#0}, core[78d0b6180d7076d4]::result::Result<rustc_codegen_ssa[79a1e11a865bc274]::back::write::CompiledModules, ()>>
  22:     0xffffac5062c8 - <<std[67261b68e235c13b]::thread::Builder>::spawn_unchecked_<<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend as rustc_codegen_ssa[79a1e11a865bc274]::traits::backend::ExtraBackendMethods>::spawn_named_thread<rustc_codegen_ssa[79a1e11a865bc274]::back::write::start_executing_work<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend>::{closure#5}, core[78d0b6180d7076d4]::result::Result<rustc_codegen_ssa[79a1e11a865bc274]::back::write::CompiledModules, ()>>::{closure#0}, core[78d0b6180d7076d4]::result::Result<rustc_codegen_ssa[79a1e11a865bc274]::back::write::CompiledModules, ()>>::{closure#1} as core[78d0b6180d7076d4]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  23:     0xffffab74f3b4 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h99a8d905b3363c45
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/alloc/src/boxed.rs:2015:9
  24:     0xffffab74f3b4 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hdc482d37839e1180
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/alloc/src/boxed.rs:2015:9
  25:     0xffffab74f3b4 - std::sys::pal::unix::thread::Thread::new::thread_start::he6d6f33e3183ae4d
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys/pal/unix/thread.rs:108:17
  26:     0xffffab590584 - start_thread
  27:     0xffffab5ffd4c - thread_start
  28:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.77.0 (aedd173a2 2024-03-17) running on aarch64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
thread 'rustc' panicked at compiler/rustc_middle/src/util/bug.rs:36:26:
/rustc/aedd173a2c086e558c2b66d3743b344f977621a7/compiler/rustc_codegen_ssa/src/back/write.rs:1956:17: panic during codegen/LLVM phase
stack backtrace:
   0:     0xffffab7439ec - std::backtrace_rs::backtrace::libunwind::trace::ha909211bcabbe3ac
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/../../backtrace/src/backtrace/libunwind.rs:104:5
   1:     0xffffab7439ec - std::backtrace_rs::backtrace::trace_unsynchronized::h8f4bf76df48e0fca
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
   2:     0xffffab7439ec - std::sys_common::backtrace::_print_fmt::h7a880f791e80e899
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:68:5
   3:     0xffffab7439ec - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h435dc28c83034320
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:44:22
   4:     0xffffab78d1e8 - core::fmt::rt::Argument::fmt::h22144fbf1e72c1e0
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/fmt/rt.rs:142:9
   5:     0xffffab78d1e8 - core::fmt::write::hd8738e3694a112d7
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/core/src/fmt/mod.rs:1120:17
   6:     0xffffab73944c - std::io::Write::write_fmt::h436b94c6697db76d
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/io/mod.rs:1846:15
   7:     0xffffab743818 - std::sys_common::backtrace::_print::ha54e28ed84d016bd
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:47:5
   8:     0xffffab743818 - std::sys_common::backtrace::print::h53c5489f7bece51d
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys_common/backtrace.rs:34:9
   9:     0xffffab746284 - std::panicking::default_hook::{{closure}}::h0cf4d4378fa545a3
  10:     0xffffab745fb8 - std::panicking::default_hook::h083086c8c3d6f7ab
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/panicking.rs:292:9
  11:     0xffffac2973f8 - <alloc[f06ece74a4fc239f]::boxed::Box<rustc_driver_impl[acc3693e9fb6d00]::install_ice_hook::{closure#0}> as core[78d0b6180d7076d4]::ops::function::Fn<(&dyn for<'a, 'b> core[78d0b6180d7076d4]::ops::function::Fn<(&'a core[78d0b6180d7076d4]::panic::panic_info::PanicInfo<'b>,), Output = ()> + core[78d0b6180d7076d4]::marker::Sync + core[78d0b6180d7076d4]::marker::Send, &core[78d0b6180d7076d4]::panic::panic_info::PanicInfo)>>::call
  12:     0xffffab74696c - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::hf2edefd0c36edede
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/alloc/src/boxed.rs:2029:9
  13:     0xffffab74696c - std::panicking::rust_panic_with_hook::h7eda7902743f1889
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/panicking.rs:785:13
  14:     0xffffb0f945c0 - std[67261b68e235c13b]::panicking::begin_panic::<alloc[f06ece74a4fc239f]::string::String>::{closure#0}
  15:     0xffffb0f94274 - std[67261b68e235c13b]::sys_common::backtrace::__rust_end_short_backtrace::<std[67261b68e235c13b]::panicking::begin_panic<alloc[f06ece74a4fc239f]::string::String>::{closure#0}, !>
  16:     0xffffac1a7bcc - std[67261b68e235c13b]::panicking::begin_panic::<alloc[f06ece74a4fc239f]::string::String>
  17:     0xffffb1004b8c - rustc_middle[a0df8d032def7fe]::util::bug::opt_span_bug_fmt::<rustc_span[a3ab065b480aa793]::span_encoding::Span>::{closure#0}
  18:     0xffffb1002bf4 - rustc_middle[a0df8d032def7fe]::ty::context::tls::with_opt::<rustc_middle[a0df8d032def7fe]::util::bug::opt_span_bug_fmt<rustc_span[a3ab065b480aa793]::span_encoding::Span>::{closure#0}, !>::{closure#0}
  19:     0xffffb1002bc4 - rustc_middle[a0df8d032def7fe]::ty::context::tls::with_context_opt::<rustc_middle[a0df8d032def7fe]::ty::context::tls::with_opt<rustc_middle[a0df8d032def7fe]::util::bug::opt_span_bug_fmt<rustc_span[a3ab065b480aa793]::span_encoding::Span>::{closure#0}, !>::{closure#0}, !>
  20:     0xffffac1a02f8 - rustc_middle[a0df8d032def7fe]::util::bug::bug_fmt
  21:     0xffffac5089d0 - <rustc_codegen_ssa[79a1e11a865bc274]::back::write::OngoingCodegen<rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend>>::join
  22:     0xffffac5bcff8 - <rustc_codegen_llvm[1c89da6189bdcd79]::LlvmCodegenBackend as rustc_codegen_ssa[79a1e11a865bc274]::traits::backend::CodegenBackend>::join_codegen
  23:     0xffffac49d394 - <rustc_interface[735d705be2a4b8df]::queries::Linker>::link
  24:     0xffffac2caed8 - rustc_interface[735d705be2a4b8df]::util::run_in_thread_with_globals::<rustc_interface[735d705be2a4b8df]::interface::run_compiler<core[78d0b6180d7076d4]::result::Result<(), rustc_span[a3ab065b480aa793]::ErrorGuaranteed>, rustc_driver_impl[acc3693e9fb6d00]::run_compiler::{closure#0}>::{closure#0}, core[78d0b6180d7076d4]::result::Result<(), rustc_span[a3ab065b480aa793]::ErrorGuaranteed>>::{closure#0}::{closure#0}
  25:     0xffffac2c9448 - std[67261b68e235c13b]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[735d705be2a4b8df]::util::run_in_thread_with_globals<rustc_interface[735d705be2a4b8df]::interface::run_compiler<core[78d0b6180d7076d4]::result::Result<(), rustc_span[a3ab065b480aa793]::ErrorGuaranteed>, rustc_driver_impl[acc3693e9fb6d00]::run_compiler::{closure#0}>::{closure#0}, core[78d0b6180d7076d4]::result::Result<(), rustc_span[a3ab065b480aa793]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[78d0b6180d7076d4]::result::Result<(), rustc_span[a3ab065b480aa793]::ErrorGuaranteed>>
  26:     0xffffac2cc1d0 - <<std[67261b68e235c13b]::thread::Builder>::spawn_unchecked_<rustc_interface[735d705be2a4b8df]::util::run_in_thread_with_globals<rustc_interface[735d705be2a4b8df]::interface::run_compiler<core[78d0b6180d7076d4]::result::Result<(), rustc_span[a3ab065b480aa793]::ErrorGuaranteed>, rustc_driver_impl[acc3693e9fb6d00]::run_compiler::{closure#0}>::{closure#0}, core[78d0b6180d7076d4]::result::Result<(), rustc_span[a3ab065b480aa793]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[78d0b6180d7076d4]::result::Result<(), rustc_span[a3ab065b480aa793]::ErrorGuaranteed>>::{closure#1} as core[78d0b6180d7076d4]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  27:     0xffffab74f3b4 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h99a8d905b3363c45
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/alloc/src/boxed.rs:2015:9
  28:     0xffffab74f3b4 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hdc482d37839e1180
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/alloc/src/boxed.rs:2015:9
  29:     0xffffab74f3b4 - std::sys::pal::unix::thread::Thread::new::thread_start::he6d6f33e3183ae4d
                               at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/library/std/src/sys/pal/unix/thread.rs:108:17
  30:     0xffffab590584 - start_thread
  31:     0xffffab5ffd4c - thread_start
  32:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: rustc 1.77.0 (aedd173a2 2024-03-17) running on aarch64-unknown-linux-gnu

note: compiler flags: -C embed-bitcode=no -C debuginfo=2 -C incremental=[REDACTED]

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
error: could not compile `river` (bin "river" test)
[Finished running. Exit status: 101]

[jamesmunns]

Maybe a dupe of https://github.com/rust-lang/rust/issues/109879 - tho this is on Linux instead of Windows, and at a different worker stage (not LTO).

[jamesmunns]

I did have this happen again, though it is very rare. One potential note is that it seems to occur when I am using cargo watch, and I hit save multiple times quickly.

It's possible what is happening is:

• Some sort of incremental compilation happens, and SOME work products are written, but others aren't
• the process is killed because cargo watch notices a new save was made
• the next compilation fails because it things something (???) should exist on disk, but doesn't (or is incomplete) due to the process being killed

Does incremental compilation have any notion of "transactions"? e.g. what if the compilation is killed?

[jamesmunns]

In the run BEFORE the error occurred, I did get this warning:

warning: variable does not need to be mutable
  --> src/main.rs:23:13
   |
23 |         let mut listeners = Listeners::new();
   |             ----^^^^^^^^^
   |             |
   |             help: remove this `mut`
   |
   = note: `#[warn(unused_mut)]` on by default

warning: error copying object file `/mnt/share/vmshare/contracts/isrg/river/source/river/target/debug/deps/river-f96f9f6f9262ae0d.26wqoch9hrpxldmx.rcgu.o` to incremental directory as `/mnt/share/vmshare/contracts/isrg/river/source/river/target/debug/incremental/river-1mpzbb0wp27rk/s-guwsu8i3hc-1glc1l2-working/26wqoch9hrpxldmx.o`: No such file or directory (os error 2)

[Running 'cargo check && cargo test && cargo run -- --config-toml ./assets/test-config.toml']                                
    Checking river v0.2.0 (/mnt/share/vmshare/contracts/isrg/river/source/river)
warning: unused variable: `listeners`
  --> src/main.rs:23:17
   |
23 |         let mut listeners = Listeners::new();
   |                 ^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_listeners`
   |

in the subsequent error:

thread 'cpy 26wqoch9hrpxldmx' panicked at /rustc/aedd173a2c086e558c2b66d3743b344f977621a7/compiler/rustc_codegen_ssa/src/back/write.rs:917:44:
no saved object file in work product
stack backtrace:
   0:     0xffff7fd439ec - std::backtrace_rs::backtrace::libunwind::trace::ha909211bcabbe3ac

[saethlin]

I did have this happen again, though it is very rare. One potential note is that it seems to occur when I am using cargo watch, and I hit save multiple times quickly.

This is great info. Maybe I can craft another test harness that provokes this crash.

[saethlin]

Does incremental compilation have any notion of "transactions"? e.g. what if the compilation is killed?

To be clear: There is supposed to be a mechanism to prevent the scenario you're reporting here. We write all our output to a staging directory then fs::rename the staging directory to the name that the next compilation is actually going to look for. The implementation was designed with this "what if the compiler is killed?" scenario in mind. It's probably just buggy.

[jamesmunns]

Also potentially worth mentioning: I'm using a VM with a shared folder for this, with:

• The host has an editor open, running Rust Analyzer
• The client VM is running cargo watch

IMO it shouldn't be THAT different than having an IDE with R-A and running cargo-watch at the same time, but just wanted to note.

[saethlin]

VM with a shared folder

Hmmmmmmmmmmmmmmmmmmmmmmm

https://github.com/rust-lang/rust/blob/4fd4797c2654977f545c9a91e2aa4e6cdbb38919/compiler/rustc_incremental/src/persist/fs.rs#L94-L104

[jamesmunns]

HMMMM indeed. For MORE detail on my setup:

• Host is MacOS Sonoma 14.3 on an M2 Max (darwin aarch64)
• Guest is Fedora 39 (linux aarch64)
• VM software is UTM v4.4.5
• The VM is using Apple Virtualization (not QEMU)
• The host is using APFS for the file system
• The guest is using virtiofs for the shared folder

I will try two things today (tho it might not proc/repro today) to see if I can retrigger the issue (it only happens once every day or two, but maybe I can try to trigger it intentionally):

1. Try to trigger it running both RA and cargo watch on the host PC only (macos)
2. Try to trigger it running ONLY cargo watch on the guest vm only, in a local (not shared) dir. Fedora uses btrfs for the / and /home mounts

In both of those cases I eliminate the "shared" folder, if I can get it to trigger in either of those cases it's probably not virtiofs' fault, if it is then it is. I'll also peek at some of the other issues to see if they are also potentially using virtualization or Weird Filesystems

[jamesmunns]

So, I wrote this crude stress testing script, sharing in case it is useful:

#!/bin/bash

set -euxo pipefail

for i in {1..1000}
do
    stime=`bc <<< "scale=2; $i/100"`
    echo "// hehe" >> ./toml.rs
    sleep $stime
done

I was able to get it to repro once (out of a couple of runs) using the VM + Host setup where it repro'd before, and it acted weird one other time (errors about "can't copy, folder doesn't exist", but no ICE).

I wasn't able to get it to repro at all in the other setups (only linux on btrfs, only macos on apfs).

I'll try it again a couple times, it doesn't proc reliably, so there's still some chance the timing is just different in the other configs.

Still definitely could be virtiofs not handling file locking or hard linking correctly.

Let me know if there's anything I can do to help with my occasionally reproing setup.

Edit, here's the "weird" errors I mention in the not-quite-repro case above:

   Compiling river v0.2.0 (/mnt/share/vmshare/contracts/isrg/river/source/river)
error: unable to copy /mnt/share/vmshare/contracts/isrg/river/source/river/target/debug/incremental/river-u700q7cd53gp/s-guxhx8y84l-96tqgp-working/5axxxb4yfcfqc0p3.o to /mnt/share/vmshare/contracts/isrg/river/source/river/target/debug/deps/river-df648408b139a4a9.5axxxb4yfcfqc0p3.rcgu.o: No such file or directory (os error 2)

error: could not compile `river` (bin "river" test) due to 1 previous error
[Finished running. Exit status: 101]
[Running 'cargo check && cargo test']
    Checking river v0.2.0 (/mnt/share/vmshare/contracts/isrg/river/source/river)
    Finished dev [unoptimized + debuginfo] target(s) in 0.52s
   Compiling river v0.2.0 (/mnt/share/vmshare/contracts/isrg/river/source/river)
error: unable to copy /mnt/share/vmshare/contracts/isrg/river/source/river/target/debug/incremental/river-u700q7cd53gp/s-guxhx9ybn5-bxtj5l-working/3c55jdxwnbidcvho.o to /mnt/share/vmshare/contracts/isrg/river/source/river/target/debug/deps/river-df648408b139a4a9.3c55jdxwnbidcvho.rcgu.o: No such file or directory (os error 2)

error: could not compile `river` (bin "river" test) due to 1 previous error
[Finished running. Exit status: 101]
[Running 'cargo check && cargo test']
    Checking river v0.2.0 (/mnt/share/vmshare/contracts/isrg/river/source/river)
    Finished dev [unoptimized + debuginfo] target(s) in 0.51s
   Compiling river v0.2.0 (/mnt/share/vmshare/contracts/isrg/river/source/river)
error: unable to copy /mnt/share/vmshare/contracts/isrg/river/source/river/target/debug/incremental/river-u700q7cd53gp/s-guxhxaykhm-1eu4zaf-working/3tluzuoovoqa9saq.o to /mnt/share/vmshare/contracts/isrg/river/source/river/target/debug/deps/river-df648408b139a4a9.3tluzuoovoqa9saq.rcgu.o: No such file or directory (os error 2)

error: could not compile `river` (bin "river" test) due to 1 previous error
[Finished running. Exit status: 101]

[saethlin]

I slapped together a little flock stress tester using the flock implementation in rustc that's used on non-Linux unices, which I think is what you have: https://github.com/saethlin/flock-stress

Can you run it in your virtiofs?

[jamesmunns]

@saethlin is there a specific way you want me to run it?

• Run two instances on the virtiofs partition on the guest linux vm
• Run one instance in the virtiofs partition on the guest linux vm and the same shared folder on the host apfs partition
• Something else?

Trying either of these it doesn't seem to crash

[saethlin]

You did everything I would have suggested. I suppose exposing a bug with such a simple test would have been too easy.

I have access to a Mac, I'll see if I can recreate your setup.