Sporadic panic compiling with low available resources

[gtema]

Sadly I can't present a code, since it happens during compiling of a pretty big project on a system with low resources (2cpu). The error is not reproducible and happens usually when system is under pressure.

Meta

rustc --version --verbose:

rustc 1.78.0 (9b00956e5 2024-04-29)
binary: rustc
commit-hash: 9b00956e56009bab2aa15d7bff10916599e3d6d6
commit-date: 2024-04-29
host: x86_64-unknown-linux-gnu
release: 1.78.0
LLVM version: 18.1.2

sccache is also being used.

Backtrace

``` thread 'rustc' panicked at compiler/rustc_metadata/src/rmeta/decoder.rs:456:36: Missing SyntaxContext 1802102 for crate "openstack_cli" stack backtrace: stack backtrace: 0: 0x7f228b0b4892 - std::backtrace_rs::backtrace::libunwind::trace::he4ee80166a02c846 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/../../backtrace/src/backtrace/libunwind.rs:105:5 1: 0x7f228b0b4892 - std::backtrace_rs::backtrace::trace_unsynchronized::h476faccf57e88641 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5 2: 0x7f228b0b4892 - std::sys_common::backtrace::_print_fmt::h430c922a77e7a59c at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:68:5 3: 0x7f228b0b4892 - ::fmt::hffecb437d922f988 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:44:22 4: 0x7f228b105a6c - core::fmt::rt::Argument::fmt::hf3df69369399bfa9 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/fmt/rt.rs:142:9 5: 0x7f228b105a6c - core::fmt::write::hd9a8d7d029f9ea1a at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/fmt/mod.rs:1153:17 6: 0x7f228b0a978f - std::io::Write::write_fmt::h0e1226b2b8d973fe at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/io/mod.rs:1843:15 7: 0x7f228b0b4664 - std::sys_common::backtrace::_print::hd2df4a083f6e69b8 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:47:5 8: 0x7f228b0b4664 - std::sys_common::backtrace::print::he907f6ad7eee41cb at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:34:9 9: 0x7f228b0b735b - std::panicking::default_hook::{{closure}}::h3926193b61c9ca9b 10: 0x7f228b0b70b3 - std::panicking::default_hook::h25ba2457dea68e65 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:292:9 11: 0x7f2287e2845d - std[e4dfbc2c3f4b09f1]::panicking::update_hook::>::{closure#0} 12: 0x7f228b0b7ac0 - as core::ops::function::Fn>::call::h022ca2c0d8c21c9e at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2034:9 13: 0x7f228b0b7ac0 - std::panicking::rust_panic_with_hook::h0ad14d90dcf5224f at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:783:13 14: 0x7f228b0b7802 - std::panicking::begin_panic_handler::{{closure}}::h4a1838a06f542647 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:657:13 15: 0x7f228b0b4d66 - std::sys_common::backtrace::__rust_end_short_backtrace::h77cc4dc3567ca904 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:171:18 16: 0x7f228b0b7534 - rust_begin_unwind at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5 17: 0x7f228b101f85 - core::panicking::panic_fmt::h940d4fd01a4b4fd1 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14 18: 0x7f22891cd967 - ::decode_syntax_context 19: 0x7f22891ceef1 - ::decode_span 20: 0x7f22891c67fb - >::decode 21: 0x7f228970423d - rustc_metadata[70192afe97828208]::rmeta::decoder::cstore_impl::provide_extern::optimized_mir 22: 0x7f2289703fad - rustc_query_impl[e4152ad88c3d6c78]::plumbing::__rust_begin_short_backtrace::> 23: 0x7f2289394c99 - rustc_query_system[475239fef39bf53f]::query::plumbing::try_execute_query::>, false, false, false>, rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt, true> 24: 0x7f2289c46f59 - rustc_query_impl[e4152ad88c3d6c78]::plumbing::force_from_dep_node::>, false, false, false>> 25: 0x7f22885cafcd - ::{closure#0} as core[836963c7c1decc11]::ops::function::FnOnce<(rustc_middle[fda44fdb505d3e7f]::ty::context::TyCtxt, rustc_query_system[475239fef39bf53f]::dep_graph::dep_node::DepNode)>>::call_once 26: 0x7f2289029055 - >::try_mark_previous_green:: 27: 0x7f2289028fbd - >::try_mark_previous_green:: 28: 0x7f2289028fbd - >::try_mark_previous_green:: 29: 0x7f2289028fbd - >::try_mark_previous_green:: 30: 0x7f2289028fbd - >::try_mark_previous_green:: 31: 0x7f2289028fbd - >::try_mark_previous_green:: 32: 0x7f2289028fbd - >::try_mark_previous_green:: 33: 0x7f2289028fbd - >::try_mark_previous_green:: 34: 0x7f2289028fbd - >::try_mark_previous_green:: 35: 0x7f2289028fbd - >::try_mark_previous_green:: 36: 0x7f2289028fbd - >::try_mark_previous_green:: 37: 0x7f2289028fbd - >::try_mark_previous_green:: 38: 0x7f2289028fbd - >::try_mark_previous_green:: 39: 0x7f2289028fbd - >::try_mark_previous_green:: 40: 0x7f2289028fbd - >::try_mark_previous_green:: 41: 0x7f2289028fbd - >::try_mark_previous_green:: 42: 0x7f2289028fbd - >::try_mark_previous_green:: 43: 0x7f2289390b19 - rustc_query_system[475239fef39bf53f]::query::plumbing::ensure_must_run::>, false, false, false>, rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt> 44: 0x7f2289390cb7 - rustc_query_impl[e4152ad88c3d6c78]::query_impl::mir_drops_elaborated_and_const_checked::get_query_incr::__rust_end_short_backtrace 45: 0x7f2289912284 - rustc_interface[ba2b6dc4c96cb491]::passes::analysis 46: 0x7f22899118e5 - rustc_query_impl[e4152ad88c3d6c78]::plumbing::__rust_begin_short_backtrace::> 47: 0x7f2289e9d375 - rustc_query_system[475239fef39bf53f]::query::plumbing::try_execute_query::>, false, false, false>, rustc_query_impl[e4152ad88c3d6c78]::plumbing::QueryCtxt, true> 48: 0x7f2289e9cf9c - rustc_query_impl[e4152ad88c3d6c78]::query_impl::analysis::get_query_incr::__rust_end_short_backtrace 49: 0x7f2289bfc814 - rustc_interface[ba2b6dc4c96cb491]::interface::run_compiler::, rustc_driver_impl[24a943716c49befe]::run_compiler::{closure#0}>::{closure#0} 50: 0x7f2289f3eb6e - std[e4dfbc2c3f4b09f1]::sys_common::backtrace::__rust_begin_short_backtrace::, rustc_driver_impl[24a943716c49befe]::run_compiler::{closure#0}>::{closure#0}, core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>> 51: 0x7f2289f3e9ca - <::spawn_unchecked_, rustc_driver_impl[24a943716c49befe]::run_compiler::{closure#0}>::{closure#0}, core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[836963c7c1decc11]::result::Result<(), rustc_span[a4517f2b2e65298c]::ErrorGuaranteed>>::{closure#1} as core[836963c7c1decc11]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} 52: 0x7f228b0c1145 - as core::ops::function::FnOnce>::call_once::h19b9e642d37e7272 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9 53: 0x7f228b0c1145 - as core::ops::function::FnOnce>::call_once::h97265befc434d3ae at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/alloc/src/boxed.rs:2020:9 54: 0x7f228b0c1145 - std::sys::pal::unix::thread::Thread::new::thread_start::h420dad5cf01a9f35 at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys/pal/unix/thread.rs:108:17 55: 0x7f2284ea91b7 - start_thread 56: 0x7f2284f2b39c - __GI___clone3 57: 0x0 - note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. error: the compiler unexpectedly panicked. this is a bug. note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md note: rustc 1.78.0 (9b00956e5 2024-04-29) running on x86_64-unknown-linux-gnu note: compiler flags: --crate-type bin -C embed-bitcode=no -C incremental=[REDACTED] -C strip=debuginfo note: some of the compiler flags provided by cargo are hidden query stack during panic: #0 [optimized_mir] optimizing MIR for `openstack_cli::block_storage::v3::default_type::delete::DefaultTypeCommand::take_action::{closure#0}` #1 [analysis] running analysis passes on this crate end of query stack there was a panic while trying to force a dep node try_mark_green dep node stack: #0 layout_of(50ce6e0430ea69d8-317261177dcface1) #1 layout_of(4b1552e6bd747bc7-77ed0138a8d8d645) #2 layout_of(4cf13c39e951339c-f5f82044666e223f) #3 layout_of(a3dee3b30cf808c3-3a6e021221241706) #4 layout_of(4f91719f6ca70abc-493fafac069970d0) #5 layout_of(3a384e266c18be90-4b0fe244ec915ac2) #6 layout_of(48ac620664a682ba-c326f5cce3af088) #7 layout_of(c18751150576cec9-e8b4b127eac2f2e7) #8 layout_of(234adbd073a917ac-5c716200772537f2) #9 layout_of(706049fc7405dae3-3b7d1f7e90c1813c) #10 layout_of(b81958a840e0f712-671f505983d84ea9) #11 layout_of(e3beb139c38be6b4-f1976512ea2c9908) #12 layout_of(5d504ab8a3fd0739-19489422ee55769e) #13 layout_of(b2bb3ff529cfa3e0-747f09f5629d4e4a) #14 layout_of(76aebd48c2d3fdb8-17f748b1343b4bbe) #15 layout_of(b5b33ee778414687-f240d2dfa2fe6a1a) #16 mir_drops_elaborated_and_const_checked(osc[a3b8]::main) end of try_mark_green dep node stack error: could not compile `openstack_cli` (bin "osc") ```

[workingjubilee]

@gtema Can you be more precise about resources? Drive space? RAM?

[workingjubilee]

I ask because the issue you mention is in the frontend, and as I understand it, the parallel frontend is not actually used by default (it is technically active but in single-threaded mode).

[gtema]

16G RAM and around 70Gb free disk (which is full in few weeks of project development, 2cpu (older i5)

[workingjubilee]

Hmmm, that's a bit low on the disk side but not so much that I would expect compiling to simply fail like that. I don't think the resources available are the problem, based on your description, as much weaker CPUs, with less RAM, and less disk, successfully compile projects.

Do you have to use ccache? The Rust compiler has its own incremental logic, and the panic here is firmly inside the "restore from previously-cached incremental compilation metadata" logic.

[gtema]

Scache is not mandatory, was just hoping to speed compilation up (since it takes around 5-7 minutes). I was suspecting the issue may lay in cache itself, since usually cargo clean helps to solve the situation for few days.

[workingjubilee]

Oh, sccache?

[gtema]

Yeah, autocorrect fixed it (facepalm) it's sccache