Const evaluation with huge arrays consumes all available memory

belovdv commented 6 months ago

Maybe, this is duplicate of #93215

Example code:

fn main() {
    let _ = &[0; 1 << 40];
    println!();
}

I expected to see this happen:

Compilation, regardless of results, consume limited amount of memory. Ideally this code should work.

Instead, this happened:

Running rustc src/main.rs on example code consumes memory, until gets Killed.

Meta

rustc --version --verbose:

rustc 1.71.1 (eb26296b5 2023-08-03)
binary: rustc
commit-hash: eb26296b556cef10fb713a38f3d16b9886080f26
commit-date: 2023-08-03
host: x86_64-unknown-linux-gnu
release: 1.71.1
LLVM version: 16.0.5

or

rustc 1.73.0-nightly (474709a9a 2023-08-03)
binary: rustc
commit-hash: 474709a9a2a74a8bcf0055fadb335d0ca0d2d939
commit-date: 2023-08-03
host: x86_64-unknown-linux-gnu
release: 1.73.0-nightly
LLVM version: 16.0.5

Backtrace from gdb

``` #0 __memset_avx2_unaligned_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:351 #1 0x00007ffff53df683 in >::write_scalar () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #2 0x00007ffff543e5e4 in >::copy_op_no_validate () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #3 0x00007ffff53a9a32 in >::statement () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #4 0x00007ffff536bda6 in rustc_const_eval::const_eval::eval_queries::eval_to_allocation_raw_provider () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #5 0x00007ffff50251f7 in rustc_query_impl::plumbing::__rust_begin_short_backtrace::> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #6 0x00007ffff5b02b0c in rustc_query_system::query::plumbing::try_execute_query::, rustc_middle::query::erase::Erased<[u8; 16]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #7 0x00007ffff5b027cd in rustc_query_impl::query_impl::eval_to_allocation_raw::get_query_non_incr::__rust_end_short_backtrace () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #8 0x00007ffff536b38e in rustc_const_eval::const_eval::eval_queries::eval_to_allocation_raw_provider () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #9 0x00007ffff50251f7 in rustc_query_impl::plumbing::__rust_begin_short_backtrace::> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #10 0x00007ffff5b02b0c in rustc_query_system::query::plumbing::try_execute_query::, rustc_middle::query::erase::Erased<[u8; 16]>>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #11 0x00007ffff5b027cd in rustc_query_impl::query_impl::eval_to_allocation_raw::get_query_non_incr::__rust_end_short_backtrace () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #12 0x00007ffff5fd5886 in >::eval_mir_constant () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #13 0x00007ffff5fcc7dc in ::visit_basic_block_data () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #14 0x00007ffff5fc8eff in ::run_lint () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #15 0x00007ffff55d5436 in _RNvCs7bEHG8lNZ4h_19rustc_mir_transform38mir_drops_elaborated_and_const_checked.llvm.12707411324951536808 () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #16 0x00007ffff4fea7ee in rustc_query_impl::plumbing::__rust_begin_short_backtrace::> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #17 0x00007ffff4fea7be in >::call_once () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #18 0x00007ffff50a09fd in rustc_query_system::query::plumbing::try_execute_query::>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #19 0x00007ffff689fd21 in rustc_query_impl::query_impl::mir_drops_elaborated_and_const_checked::get_query_non_incr::__rust_end_short_backtrace () --Type for more, q to quit, c to continue without paging-- from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #20 0x00007ffff63c6c5d in ::time::<(), rustc_interface::passes::analysis::{closure#2}> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #21 0x00007ffff63c4e66 in rustc_interface::passes::analysis () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #22 0x00007ffff640a8fa in rustc_query_impl::plumbing::__rust_begin_short_backtrace::> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #23 0x00007ffff640a8e9 in _RNvYNCNvNtNtCs9t0W4sfjaFR_16rustc_query_impl10query_impl8analysis13dynamic_querys0_0INtNtNtCscS4HM4tpJt3_4core3ops8function6FnOnceTNtNtNtCsh2FAgqFZnx3_12rustc_middle2ty7context6TyCtxtuEE9call_onceBa_.llvm.10073415029295238485 () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #24 0x00007ffff65c7fe8 in rustc_query_system::query::plumbing::try_execute_query::>, false, false, false>, rustc_query_impl::plumbing::QueryCtxt, false> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #25 0x00007ffff65c7db9 in rustc_query_impl::query_impl::analysis::get_query_non_incr::__rust_end_short_backtrace () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #26 0x00007ffff6152392 in >::enter::, rustc_driver_impl::run_compiler::{closure#1}::{closure#2}::{closure#4}> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #27 0x00007ffff6151475 in ::enter::, rustc_span::ErrorGuaranteed>> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #28 0x00007ffff614f5c4 in rustc_span::set_source_map::, rustc_interface::interface::run_compiler, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}::{closure#0}> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #29 0x00007ffff614f1a1 in >::set::, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #30 0x00007ffff614e796 in std::sys_common::backtrace::__rust_begin_short_backtrace::, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#0}::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>> () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #31 0x00007ffff67a64c5 in <::spawn_unchecked_, rustc_driver_impl::run_compiler::{closure#1}>::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#0}::{closure#0}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#1} as core::ops::function::FnOnce<()>>::call_once::{shim:vtable#0} () from /home/$USER/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/../lib/librustc_driver-92234e9dc706375c.so #32 0x00007ffff3d777b5 in alloc::boxed::{impl#47}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1985 #33 alloc::boxed::{impl#47}::call_once<(), alloc::boxed::Box, alloc::alloc::Global>, alloc::alloc::Global> () at library/alloc/src/boxed.rs:1985 #34 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108 #35 0x00007ffff3a94ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442 #36 0x00007ffff3b26a40 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 ```

Noratrieb commented 6 months ago

This is a very big constant, more memory than you have. How would you expect this to work?

scottmcm commented 6 months ago

Compilation, regardless of results, consume limited amount of memory.

Note that this, in general, cannot be true. You can write and evaluate const fns that will need arbitrarily-high amounts of memory and compute.

workingjubilee commented 6 months ago

There doesn't seem to be anything we can do here. Constants are evaluated by an interpreter, thus when you write them down, you actually allocate the memory and run a program at compiletime. That is, compiletime is just another runtime. We can make the interpreter more efficient on some cases but this is genuinely insoluble in the general case.

Closing.

rust-lang / rust

Const evaluation with huge arrays consumes all available memory #123805

Meta