ICE: MemDecoder exhausted

erikh commented 10 months ago

Code

I do not have a minimum verifiable example. It's hard to decipher this backtrace. However, I have https://github.com/erikh/megalock/commit/daed9c04b202a34c18570779badfff2c641c9331 which should demonstrate this error on 1.75.0.

EDIT: sorry, I had the url syntax wrong. That link should point at the correct commit.

Error output

thread 'rustc' panicked at compiler/rustc_serialize/src/opaque.rs:257:9:
MemDecoder exhausted
stack backtrace:
   0:     0x7fc9fbea462c - std::backtrace_rs::backtrace::libunwind::trace::ha637c64ce894333a
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/../../backtrace/src/backtrace/libunwi
nd.rs:104:5
   1:     0x7fc9fbea462c - std::backtrace_rs::backtrace::trace_unsynchronized::h47f62dea28e0c88d
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/../../backtrace/src/backtrace/mod.rs:
66:5
   2:     0x7fc9fbea462c - std::sys_common::backtrace::_print_fmt::h9eef0abe20ede486
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:67:5
   3:     0x7fc9fbea462c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hed7f999df88cc644
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:44:22
   4:     0x7fc9fbef7630 - core::fmt::rt::Argument::fmt::h1539a9308b8d058d
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/fmt/rt.rs:142:9
   5:     0x7fc9fbef7630 - core::fmt::write::h3a39390d8560d9c9
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/fmt/mod.rs:1120:17
   6:     0x7fc9fbe9854f - std::io::Write::write_fmt::h5fc9997dfe05f882
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/io/mod.rs:1762:15
   7:     0x7fc9fbea4414 - std::sys_common::backtrace::_print::h894006fb5c6f3d45
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:47:5
   8:     0x7fc9fbea4414 - std::sys_common::backtrace::print::h23a2d212c6fff936
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:34:9
   9:     0x7fc9fbea70a7 - std::panicking::default_hook::{{closure}}::h8a1d2ee00185001a
  10:     0x7fc9fbea6e0f - std::panicking::default_hook::h6038f2eba384e475
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:292:9
  11:     0x7fc9f8d37190 - std[409886f6357001f0]::panicking::update_hook::<alloc[c1b021ad36e35877]::boxed::Box<rustc_driver_impl[7d23c5
715ff089db]::install_ice_hook::{closure#0}>>::{closure#0}
  12:     0x7fc9fbea77e8 - <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call::h1f8f335eaa9cfaee
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2021:9
  13:     0x7fc9fbea77e8 - std::panicking::rust_panic_with_hook::h2b5517d590cab22e
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:783:13
  14:     0x7fc9fbea7509 - std::panicking::begin_panic_handler::{{closure}}::h233112c06e0ef43e
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:649:13
  15:     0x7fc9fbea4af6 - std::sys_common::backtrace::__rust_end_short_backtrace::h6e893f24d7ebbff8
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:170:18
  16:     0x7fc9fbea72a2 - rust_begin_unwind
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
  17:     0x7fc9fbef3d15 - core::panicking::panic_fmt::hbf0e066aabfa482c
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
  18:     0x7fc9f95eaf9a - <rustc_serialize[5d0f213b508bf701]::opaque::MemDecoder>::decoder_exhausted
  19:     0x7fc9fb03c696 - <rustc_middle[aca4860da4e5a967]::query::on_disk_cache::OnDiskCache>::new
  20:     0x7fc9fae3caae - rustc_incremental[8fad7206137c3013]::persist::load::load_query_result_cache
  21:     0x7fc9fae3ccc8 - rustc_interface[fbb0cb4be6c0ba34]::passes::create_global_ctxt
  22:     0x7fc9faf201df - <rustc_interface[fbb0cb4be6c0ba34]::queries::Queries>::global_ctxt
  23:     0x7fc9fae06dc0 - rustc_interface[fbb0cb4be6c0ba34]::interface::run_compiler::<core[21cdcf8e8af4c2d9]::result::Result<(), rust
c_span[3d5dc97049ad8d50]::ErrorGuaranteed>, rustc_driver_impl[7d23c5715ff089db]::run_compiler::{closure#1}>::{closure#0}
  24:     0x7fc9fae0215b - std[409886f6357001f0]::sys_common::backtrace::__rust_begin_short_backtrace::<rustc_interface[fbb0cb4be6c0ba3
4]::util::run_in_thread_with_globals<rustc_interface[fbb0cb4be6c0ba34]::interface::run_compiler<core[21cdcf8e8af4c2d9]::result::Result<
(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>, rustc_driver_impl[7d23c5715ff089db]::run_compiler::{closure#1}>::{closure#0}, core[
21cdcf8e8af4c2d9]::result::Result<(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[21cdcf8e8af4c2d9]
::result::Result<(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>>
  25:     0x7fc9fae01fb3 - <<std[409886f6357001f0]::thread::Builder>::spawn_unchecked_<rustc_interface[fbb0cb4be6c0ba34]::util::run_in_
thread_with_globals<rustc_interface[fbb0cb4be6c0ba34]::interface::run_compiler<core[21cdcf8e8af4c2d9]::result::Result<(), rustc_span[3d
5dc97049ad8d50]::ErrorGuaranteed>, rustc_driver_impl[7d23c5715ff089db]::run_compiler::{closure#1}>::{closure#0}, core[21cdcf8e8af4c2d9]
::result::Result<(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>>::{closure#0}::{closure#0}, core[21cdcf8e8af4c2d9]::result::Result<
(), rustc_span[3d5dc97049ad8d50]::ErrorGuaranteed>>::{closure#1} as core[21cdcf8e8af4c2d9]::ops::function::FnOnce<()>>::call_once::{shi
m:vtable#0}
  26:     0x7fc9fbeb16a5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::hc7eafaff61e32df9
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
  27:     0x7fc9fbeb16a5 - <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once::h6ba4a5de48dd2304
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/alloc/src/boxed.rs:2007:9
  28:     0x7fc9fbeb16a5 - std::sys::unix::thread::Thread::new::thread_start::he469335aef763e45
                               at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys/unix/thread.rs:108:17
  29:     0x7fc9f60aa9eb - <unknown>
  30:     0x7fc9f612e7cc - <unknown>
  31:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

Backtrace

The backtrace comes from rustc itself.

Just doing my part, looks like my repo did something to hose the compiler. Next steps I'm happy to help with, but the repo is probably the easiest way to investigate this independently. I'm sorry I'm not more helpful.

erikh commented 10 months ago

Since I don't know if it's relevant, here's a ldd megalock on a recent optimized bin because it does link against numerous C shared libraries:

    linux-vdso.so.1 (0x00007fff2f555000)
    libxkbcommon.so.0 => /usr/lib/libxkbcommon.so.0 (0x00007f97388a4000)
    libxkbcommon-x11.so.0 => /usr/lib/libxkbcommon-x11.so.0 (0x00007f973889b000)
    libcairo.so.2 => /usr/lib/libcairo.so.2 (0x00007f9738762000)
    libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00007f9738737000)
    libxcb-randr.so.0 => /usr/lib/libxcb-randr.so.0 (0x00007f9738725000)
    libxcb-image.so.0 => /usr/lib/libxcb-image.so.0 (0x00007f973871f000)
    libxcb-xrm.so.0 => /usr/lib/libxcb-xrm.so.0 (0x00007f9738715000)
    libxcb-xkb.so.1 => /usr/lib/libxcb-xkb.so.1 (0x00007f97386f6000)
    libxcb-util.so.1 => /usr/lib/libxcb-util.so.1 (0x00007f97386ef000)
    libpam.so.0 => /usr/lib/libpam.so.0 (0x00007f97386de000)
    libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f97386b9000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007f97385cc000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f97383e8000)
    libz.so.1 => /usr/lib/libz.so.1 (0x00007f97383ce000)
    libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007f9738394000)
    libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x00007f9738344000)
    libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007f9738276000)
    libX11.so.6 => /usr/lib/libX11.so.6 (0x00007f9738133000)
    libXext.so.6 => /usr/lib/libXext.so.6 (0x00007f973811c000)
    libXrender.so.1 => /usr/lib/libXrender.so.1 (0x00007f973810f000)
    libxcb-render.so.0 => /usr/lib/libxcb-render.so.0 (0x00007f9738100000)
    libxcb-shm.so.0 => /usr/lib/libxcb-shm.so.0 (0x00007f97380fb000)
    libpixman-1.so.0 => /usr/lib/libpixman-1.so.0 (0x00007f973804e000)
    libXau.so.6 => /usr/lib/libXau.so.6 (0x00007f9738049000)
    libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x00007f973803f000)
    libaudit.so.1 => /usr/lib/libaudit.so.1 (0x00007f9738015000)
    /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f97389d7000)
    libexpat.so.1 => /usr/lib/libexpat.so.1 (0x00007f9737fea000)
    libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007f9737fd7000)
    libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0x00007f9737ec9000)
    libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x00007f9737eb8000)
    libcap-ng.so.0 => /usr/lib/libcap-ng.so.0 (0x00007f9737eb0000)
    libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007f9737d64000)
    libgraphite2.so.3 => /usr/lib/libgraphite2.so.3 (0x00007f9737d41000)
    libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x00007f9737d1e000)
    libpcre2-8.so.0 => /usr/lib/libpcre2-8.so.0 (0x00007f9737c81000)

erikh commented 10 months ago

Upgrading to the latest version of my crates, proc_macro2 was in there as well as syn, seemed to have corrected the error. Could also be lurking somewhere in the build cache. If you want that I suppose I can provide it somewhere.

Not sure what to do with this bug now, but feel free to close it if you no longer feel it's relevant.

Noratrieb commented 10 months ago

@saethlin does your PR fix this?

saethlin commented 10 months ago

Hm, it looks like the problem here is that the OnDiskCache footer is off the end of the file. When that happens due to out-of-disk, people tend to notice that they are out of disk. @erikh Am I correct in assuming you had some disk space available when this crash happened?

If this is an out-of-disk, it was probably "solved" by https://github.com/rust-lang/rust/pull/117301 then will be actually solved by https://github.com/rust-lang/rust/pull/119510. But if isn't an out-of-disk situation, this is probably an incremental compilation bug. The build directory state at the time the crash happened would be useful, but if the crash doesn't recur anymore on your end then the problematic state in the build directory is probably gone. In that case, this issue is still useful to say that something is wrong here, but that's kind of all.

erikh commented 10 months ago

@saethlin correct, I had about 900GB free on this mountpoint at the time of compilation.

erikh commented 10 months ago

It does also make sense that my build cache was corrupt; refreshing crates seemed to correct the issue. Maybe a truncated object write or something? Just throwing things at the wall here.

Either way, my personal issue is resolved, but I figure this is probably worth keeping around for resolution on a more reasonable time frame.

saethlin commented 10 months ago

Crashes like this in incremental compilation are usually due to subtle internal things like a Hash impl used in the incremental compilation cache that accidentally has a high collision rate in some rare situation.

erikh commented 10 months ago

Ah, interesting. So heat death of the universe before this bug probably gets closed, eh? :)

I'll leave ya to it.

rust-lang / rust

ICE: MemDecoder exhausted #119511

Code

Meta

Error output