Compiler using over 20GiB memory

[davidspies]

I have a project where cargo build runs for about 5 minutes before spitting out:

error: could not compile `boolsatr`

Caused by:
  process didn't exit successfully: `rustc --crate-name boolsatr --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=255257b6ec403593 -C extra-filename=-255257b6ec403593 --out-dir /home/david/projects/boolsatr/target/debug/deps -C incremental=/home/david/projects/boolsatr/target/debug/incremental -L dependency=/home/david/projects/boolsatr/target/debug/deps --extern dc2=/home/david/projects/boolsatr/target/debug/deps/libdc2-2d8e237aaa1a2253.rlib --extern lazy_fields=/home/david/projects/boolsatr/target/debug/deps/liblazy_fields-448c098331ae6c36.rlib --extern rand=/home/david/projects/boolsatr/target/debug/deps/librand-e3fc23702a4e7632.rlib --extern rand_pcg=/home/david/projects/boolsatr/target/debug/deps/librand_pcg-8bccd6915bda7571.rlib` (signal: 9, SIGKILL: kill)
The terminal process "/bin/bash '-c', 'cargo build'" terminated with exit code: 101.

(cargo check runs without issue)

I assume the compiler's getting stuck in a loop somewhere and cargo is killing it after some timeout, but to produce a MWE I need some idea of which bit of code is causing it. How can I figure this out?

Meta

rustc --version --verbose:

rustc 1.52.0-nightly (07194ffcd 2021-02-10)
binary: rustc
commit-hash: 07194ffcd25b0871ce560b9f702e52db27ac9f77
commit-date: 2021-02-10
host: x86_64-unknown-linux-gnu
release: 1.52.0-nightly
LLVM version: 11.0.1

Adding RUST_BACKTRACE=1 doesn't change the error message

[jyn514]

You can find where the time is being spent with -Z time-passes, which prints the times eagerly. Specifically cargo rustc --bin -- -Z time-passes should work here.

[jyn514]

Also, SIGKILL usually means another process killed it - are you running out of memory maybe?

[davidspies]

yup That's what's happening. I was looking at top and rustc seems to be trying to use more memory than I have on my laptop (32 GiB). I think the compiler needing more than 32 GiB memory for a ~5000 LOC project probably still qualifies as a bug.

[jyn514]

Can you try to make an MCVE? If not, can you link the project that's using so much memory?

[davidspies]

I tried -Z time-passes but it doesn't print out the step that's using all the memory because it's killed before finishing. I tried killing it with ctrl-C hoping that it would print out something on SIGTERM rather than SIGKILL but still nothing.

However, you can see an earlier step where it used 7 GiB. So it might just be another instantiation of that:

$ cargo rustc --bin boolsatr -- -Z time-passes
   Compiling boolsatr v0.1.0 (/home/david/projects/boolsatr)
time:   0.006; rss:   50MB ->   52MB (   +1MB)  parse_crate
time:   0.001; rss:   52MB ->   52MB (   +0MB)  attributes_injection
time:   0.004; rss:   52MB ->   53MB (   +2MB)  incr_comp_prepare_session_directory
time:   0.001; rss:   53MB ->   53MB (   +0MB)  incr_comp_garbage_collect_session_directories
time:   0.001; rss:   53MB ->   53MB (   +0MB)  recursion_limit
time:   0.000; rss:   53MB ->   53MB (   +0MB)  plugin_loading
time:   0.000; rss:   53MB ->   53MB (   +0MB)  plugin_registration
time:   0.002; rss:   55MB ->   55MB (   +0MB)  crate_injection
time:   0.228; rss:   55MB ->   80MB (  +25MB)  expand_crate
time:   0.000; rss:   80MB ->   80MB (   +0MB)  check_unused_macros
time:   0.229; rss:   55MB ->   80MB (  +25MB)  macro_expand_crate
time:   0.000; rss:   80MB ->   80MB (   +0MB)  maybe_building_test_harness
time:   0.001; rss:   80MB ->   80MB (   +0MB)  AST_validation
time:   0.000; rss:   80MB ->   80MB (   +0MB)  maybe_create_a_macro_crate
time:   0.001; rss:   80MB ->   80MB (   +0MB)  finalize_imports
time:   0.001; rss:   80MB ->   81MB (   +0MB)  finalize_macro_resolutions
time:   0.023; rss:   81MB ->   84MB (   +3MB)  late_resolve_crate
time:   0.001; rss:   84MB ->   84MB (   +0MB)  resolve_check_unused
time:   0.000; rss:   84MB ->   84MB (   +0MB)  resolve_report_errors
time:   0.000; rss:   84MB ->   84MB (   +0MB)  resolve_postprocess
time:   0.026; rss:   80MB ->   84MB (   +3MB)  resolve_crate
time:   0.001; rss:   84MB ->   84MB (   +0MB)  complete_gated_feature_checking
time:   0.276; rss:   53MB ->   84MB (  +30MB)  configure_and_expand
time:   0.001; rss:   84MB ->   84MB (   +0MB)  prepare_outputs
time:   0.002; rss:   84MB ->   84MB (   +0MB)  blocked_on_dep_graph_loading
time:   0.009; rss:   84MB ->   87MB (   +3MB)  hir_lowering
time:   0.001; rss:   87MB ->   87MB (   +0MB)  early_lint_checks
time:   0.005; rss:   87MB ->   89MB (   +2MB)  setup_global_ctxt
time:   0.012; rss:   87MB ->   89MB (   +2MB)  create_global_ctxt
time:   0.003; rss:   92MB ->   92MB (   +0MB)  looking_for_entry_point
time:   0.001; rss:   92MB ->   92MB (   +0MB)  looking_for_plugin_registrar
time:   0.000; rss:   92MB ->   92MB (   +0MB)  looking_for_derive_registrar
time:   0.038; rss:   92MB ->   99MB (   +7MB)  misc_checking_1
time:   0.020; rss:   99MB ->  102MB (   +3MB)  type_collecting
time:   0.001; rss:  102MB ->  102MB (   +0MB)  impl_wf_inference
time:   0.000; rss:  139MB ->  139MB (   +0MB)  unsafety_checking
time:   0.000; rss:  139MB ->  139MB (   +0MB)  orphan_checking
time:   0.166; rss:  102MB ->  139MB (  +37MB)  coherence_checking
time:   0.043; rss:  139MB ->  144MB (   +5MB)  wf_checking
time:   0.003; rss:  144MB ->  144MB (   +0MB)  item_types_checking
time:   0.177; rss:  144MB ->  155MB (  +12MB)  item_bodies_checking
time:   0.409; rss:   99MB ->  155MB (  +57MB)  type_check_crate
time:   0.004; rss:  155MB ->  156MB (   +1MB)  match_checking
time:   0.004; rss:  156MB ->  158MB (   +2MB)  liveness_and_intrinsic_checking
time:   0.008; rss:  155MB ->  158MB (   +3MB)  misc_checking_2
time:   0.197; rss:  158MB ->  176MB (  +18MB)  MIR_borrow_checking
time:   0.000; rss:  176MB ->  176MB (   +0MB)  MIR_effect_checking
time:   0.000; rss:  176MB ->  176MB (   +0MB)  layout_testing
warning: associated function is never used: `dump_stats`
   --> src/solver/mod.rs:101:12
    |
101 |     pub fn dump_stats(&mut self, _graph_file: &mut File) -> Result<(), io::Error> {
    |            ^^^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default

time:   0.003; rss:  177MB ->  177MB (   +1MB)  death_checking
time:   0.001; rss:  177MB ->  177MB (   +0MB)  unused_lib_feature_checking
time:   0.003; rss:  177MB ->  179MB (   +2MB)  crate_lints
time:   0.002; rss:  179MB ->  179MB (   +0MB)  module_lints
time:   0.005; rss:  177MB ->  179MB (   +2MB)  lint_checking
time:   0.008; rss:  179MB ->  179MB (   +0MB)  privacy_checking_modules
time:   0.020; rss:  176MB ->  179MB (   +3MB)  misc_checking_3
time:   0.003; rss:  179MB ->  180MB (   +1MB)  monomorphization_collector_root_collections
time:  93.116; rss:  180MB ->  254MB (  +74MB)  monomorphization_collector_graph_walk
time:  45.622; rss:  254MB -> 7394MB (+7140MB)  partition_and_assert_distinct_symbols
time:   0.001; rss: 7395MB -> 7395MB (   +0MB)  write_allocator_module
time:   0.000; rss: 7395MB -> 7395MB (   +0MB)  find_cgu_reuse
time:   0.000; rss: 5826MB -> 5826MB (   +0MB)  LLVM_module_optimize_function_passes(2yqwcofw48oip3q)
time:   0.000; rss: 5826MB -> 5826MB (   +0MB)  LLVM_module_optimize_module_passes(2yqwcofw48oip3q)
time:   0.002; rss: 5826MB -> 5827MB (   +1MB)  LLVM_module_optimize_function_passes(5054vo9ldvu6shng)
time:   0.037; rss: 5827MB -> 5832MB (   +5MB)  LLVM_module_optimize_module_passes(5054vo9ldvu6shng)
time:   0.000; rss: 5859MB -> 5859MB (   +0MB)  LLVM_module_optimize_function_passes(483dpiq9qkl9z8jb)
time:   0.000; rss: 5859MB -> 5859MB (   +0MB)  LLVM_module_optimize_module_passes(483dpiq9qkl9z8jb)
time:   0.001; rss: 5859MB -> 5860MB (   +1MB)  LLVM_module_optimize_function_passes(2o3q8cccyyjhbuub)
time:   0.028; rss: 5860MB -> 5864MB (   +4MB)  LLVM_module_optimize_module_passes(2o3q8cccyyjhbuub)
time:   0.000; rss: 5905MB -> 5905MB (   +0MB)  LLVM_module_optimize_function_passes(51uxob6iog635dyd)
time:   0.000; rss: 5905MB -> 5905MB (   +0MB)  LLVM_module_optimize_module_passes(51uxob6iog635dyd)
time:   0.002; rss: 5905MB -> 5906MB (   +1MB)  LLVM_module_optimize_function_passes(4rj8oul8m6jyjmj1)
time:   0.044; rss: 5906MB -> 5914MB (   +8MB)  LLVM_module_optimize_module_passes(4rj8oul8m6jyjmj1)
time:   0.000; rss: 5921MB -> 5921MB (   +0MB)  LLVM_module_optimize_function_passes(42r33rw64tassjdf)
time:   0.000; rss: 5921MB -> 5921MB (   +0MB)  LLVM_module_optimize_function_passes(8q5lsqmu640v7oj)
time:   0.000; rss: 5921MB -> 5921MB (   +0MB)  LLVM_module_optimize_module_passes(8q5lsqmu640v7oj)
time:   0.009; rss: 5921MB -> 5922MB (   +2MB)  LLVM_module_optimize_module_passes(42r33rw64tassjdf)
time:   0.000; rss: 11669MB -> 11669MB (   +0MB)    LLVM_module_optimize_function_passes(2wu69c6uhy5ywqpg)
time:   0.000; rss: 11669MB -> 11669MB (   +0MB)    LLVM_module_optimize_module_passes(2wu69c6uhy5ywqpg)
time:   0.001; rss: 11669MB -> 11669MB (   +0MB)    LLVM_module_optimize_function_passes(z221iw9si33ns95)
time:   0.235; rss: 11669MB -> 11695MB (  +26MB)    LLVM_module_optimize_module_passes(z221iw9si33ns95)
time:   0.000; rss: 14330MB -> 14331MB (   +1MB)    LLVM_module_optimize_function_passes(9wflittwaemudxk)
time:   0.000; rss: 14331MB -> 14331MB (   +0MB)    LLVM_module_optimize_function_passes(389lxyptdtznlxxi)
time:   0.000; rss: 14331MB -> 14331MB (   +0MB)    LLVM_module_optimize_module_passes(389lxyptdtznlxxi)
time:   0.009; rss: 14331MB -> 14359MB (  +28MB)    LLVM_module_optimize_module_passes(9wflittwaemudxk)
^C  Building [========================>  ] 14/15: boolsatr(bin)                                                                                                                                                                           

[davidspies]

Okay I have a MCVE (where M stands for "Much smaller")

This uses over 20 GiB to compile

code

`Cargo.toml`: ``` [dependencies] dc2 = {git = "https://github.com/davidspies/dc2", branch = "main"} lazy_fields = {git = "https://github.com/davidspies/lazy-fields", branch = "main"} ``` `Cargo.lock`: ``` # This file is automatically @generated by Cargo. # It is not intended for manual editing. version = 3 [[package]] name = "boolsatr" version = "0.1.0" dependencies = [ "dc2", "lazy_fields", ] [[package]] name = "dc2" version = "0.1.0" source = "git+https://github.com/davidspies/dc2?branch=main#9e1fb2ea481ad9b8774a0c3a9d29460fec980a2b" [[package]] name = "lazy_fields" version = "0.1.0" source = "git+https://github.com/davidspies/lazy-fields?branch=main#79d2e45d2a43972938fd80048c9376dfeb0a029b" ``` `main.rs`: ```rust mod collection_ops { use crate::tuple::{snd, swap}; use dc2::{key::Key, monoid::Monoid, Collection, Op, Relation}; use std::ops::Mul; pub trait SemiJoinOn<'a, D1: Key, D2, R: Monoid> { fn semijoin_on D2 + 'static, C2: Op>( self, other: Relation<'a, C2>, f: F, ) -> Collection<'a, D1, R>; } impl<'a, D1: Key, D2: Key, R: Monoid + Mul, C: Op> SemiJoinOn<'a, D1, D2, R> for Relation<'a, C> { fn semijoin_on D2 + 'static, C2: Op>( self, other: Relation<'a, C2>, f: F, ) -> Collection<'a, D1, R> { self.map(move |val| (f(&val), val)) .semijoin(other) .map(snd) .collect() } } pub trait SemiJoinOnSnd<'a, D1: Key, D2: Key, R: Monoid> { fn semijoin_on_snd>( self, other: Relation<'a, C2>, ) -> Collection<'a, (D1, D2), R>; } impl<'a, D1: Key, D2: Key, R: Monoid + Mul, C: Op> SemiJoinOnSnd<'a, D1, D2, R> for Relation<'a, C> { fn semijoin_on_snd>( self, other: Relation<'a, C2>, ) -> Collection<'a, (D1, D2), R> { self.map(swap).semijoin(other).map(swap).collect() } } } mod ops { use std::hash::{Hash, Hasher}; use std::ops::Deref; use std::rc::Rc; #[derive(Debug)] pub struct RcRaw(pub Rc); impl Clone for RcRaw { fn clone(&self) -> Self { RcRaw(Rc::clone(&self.0)) } } impl PartialEq for RcRaw { fn eq(&self, other: &Self) -> bool { Rc::ptr_eq(&self.0, &other.0) } } impl Eq for RcRaw {} impl Hash for RcRaw { fn hash(&self, state: &mut H) { Rc::as_ptr(&self.0).hash(state); } } impl Deref for RcRaw { type Target = T; fn deref(&self) -> &T { Rc::deref(&self.0) } } } mod primitives { use std::cmp::Ordering; use std::ops::Not; #[derive(Debug, PartialEq, Eq, Clone, Copy, Hash)] pub struct Assig(isize); #[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone, Copy, Hash)] pub struct Var(isize); pub type MicroLevel = usize; impl Assig { pub fn var(self) -> Var { Var(self.0.abs()) } } impl Not for Assig { type Output = Assig; fn not(self) -> Self::Output { Assig(-self.0) } } impl PartialOrd for Assig { fn partial_cmp(&self, other: &Self) -> Option { Some(self.cmp(other)) } } impl Ord for Assig { fn cmp(&self, other: &Self) -> Ordering { match self.0.abs().cmp(&other.0.abs()) { Ordering::Equal => self.0.cmp(&other.0), res => res, } } } #[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Clone, Copy, Hash)] pub struct RuleIndex(usize); } mod program { use crate::primitives::{Assig, RuleIndex, Var}; use dc2::Collection; #[derive(Clone)] pub struct RulesCollections { pub rule_index: Collection<'static, RuleIndex>, pub rule: Collection<'static, (RuleIndex, Assig)>, pub vars: Collection<'static, Var>, } } mod solver { mod binary { use super::learnt::RefRule; use super::primitives::DecisionLevel; use super::SolverCollections; use crate::collection_ops::{SemiJoinOn, SemiJoinOnSnd}; use crate::primitives::{Assig, RuleIndex, Var}; use crate::tuple::{fst, snd, swap}; use dc2::key::Key; use dc2::map::{AssertOnes, VecMap}; use dc2::{ Arrangement, Collection, CreationContext, Input, MapMapArrangement, MappingArrangement, }; use std::collections::hash_map::DefaultHasher; use std::collections::{BTreeMap, HashMap}; use std::hash::{Hash, Hasher}; pub struct Binary { pub binary_input: Input<((Assig, Assig), (RefRule, DecisionLevel))>, pub binary_output: MapMapArrangement, pub binary_by_level: MapMapArrangement, pub closure: MappingArrangement<(Assig, Assig), usize>, pub self_implied: Arrangement< (usize, (Var, Assig)), isize, BTreeMap>>, >, pub other_un_impls: MapMapArrangement, pub other_bin_impls: MapMapArrangement<(Assig, Assig), RuleIndex>, } #[derive(Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] pub enum Path { Direct(RuleIndex), Indirect(RefRule), } impl SolverCollections<'_> { pub fn make_binary(&self) -> Binary { let active = self.active.get(); let rule_sizes = self.rule_sizes.get().clone(); let binary_rule_inds = rule_sizes .filter(|&(_key, count)| count == 2) .map(fst) .named("binary_rule_inds"); let binary_rules = active .rule .clone() .semijoin(binary_rule_inds) .reduce(|_i, xs: &HashMap| { let mut iter = xs.iter().assert_ones(); let &x = iter.next().expect("No assigs"); let &y = iter.next().expect("Only 1 assig"); assert!(iter.next().is_none(), "More than 2 assigs"); VecMap::new(vec![((!x, y), 1), ((!y, x), 1)]) }) .map(|(i, imp)| (imp, Path::Direct(i))) .named("binary_rules"); let (binary_input, binary_manual) = self.context.borrow().create_input(); let binary_manual = binary_manual.split().named("binary_manual"); let binary_by_level: MapMapArrangement = binary_manual .clone() .map(|(xy, (i, dl))| (dl, (xy, i))) .dynamic() .named("binary_by_level") .get_arrangement(&self.context.borrow()); let verts = self.rem_lits.get(); let binary = binary_rules .concat( binary_manual .semijoin_on(verts.clone(), |&((x, _), _)| x) .semijoin_on(verts.clone(), |&((_, y), _)| y) .map(|(xy, (i, _))| (xy, Path::Indirect(i))), ) .split() .named("binary"); let closure_raw = transitive_closure( &mut self.context.borrow_mut(), verts.clone(), binary.clone().map(fst).collect(), ) .named("closure_raw"); let (closure, closure_output) = closure_raw.assert_1to1_with_output(&self.context.borrow()); let closure = closure.split().named("closure"); let self_implied = closure .clone() .flat_map(|((x, y), d)| { if y == !x { Some((d, (y.var(), y))) } else { None } }) .dynamic() .named("self_implied"); let rule_sizes = self.rule_sizes.get(); let chosen_rule_index = rule_sizes .clone() .flat_map(|(i, rs)| if rs >= 3 { Some(i) } else { None }) .named("chosen_rule_index"); let chosen_rule = active .rule .clone() .semijoin(chosen_rule_index) .split() .named("chosen_rule"); let pairings = chosen_rule .clone() .map(move |(i, x)| { let mut h = DefaultHasher::new(); (i, x).hash(&mut h); (i, (h.finish(), x)) }) .reduce(|_, xs: &BTreeMap<(u64, Assig), isize>| { let mut iter = xs.iter().assert_ones(); let &(_, x) = iter.next().expect("Empty rule"); let &(_, y) = iter.next().expect("Unary rule"); let &(_, z) = iter.next().expect("Binary rule"); VecMap::new(vec![((x, y), 1), ((x, z), 1), ((y, z), 1)]) }) .split() .named("pairings"); let implied = closure.map(fst).split().named("implied"); let both_implied = pairings .clone() .map(snd) .triangles(implied.clone(), implied.clone()) .map(|(x, y, z)| ((x, y), z)) .named("both_implied"); let candidates = pairings .map(swap) .join(both_implied) .map(snd) .distinct() .named("candidates"); let impl_count = chosen_rule .clone() .join(candidates) .dynamic() // TODO Why does removing this cause the compiler to hang? .semijoin_on_snd(implied.clone()) .map(|(i, (_, y))| (i, y)) .counts() .map(|((i, y), c)| { assert!(c >= 2); (i, (y, c)) }) .named("impl_count"); let critical = impl_count .join(rule_sizes.clone()) .flat_map(|(i, ((y, c), rs))| if c >= rs - 1 { Some((i, y)) } else { None }) .split() .named("critical"); let new_binary = chosen_rule .join(critical.clone()) .map(swap) .antijoin(implied.clone()) .map(|((x, y), i)| ((!x, y), i)) .split() .named("new_binary"); let new_unary = critical .map(swap) .concat(new_binary.clone().map(|((_, y), i)| (y, i)).negate()) .named("new_unary"); let context = self.context.borrow(); Binary { binary_input, binary_output: binary .map(|((x, y), i)| (x, (y, i))) .dynamic() .named("binary_output") .get_arrangement(&context), binary_by_level, closure: Box::new(closure_output), self_implied: self_implied.get_arrangement(&context), other_un_impls: new_unary .map(|(x, i)| (x.var(), (x, i))) .dynamic() .named("other_un_impls") .get_arrangement(&context), other_bin_impls: new_binary .antijoin(implied) .dynamic() .named("other_bin_impls") .get_arrangement(&context), } } } fn transitive_closure( context: &mut CreationContext, verts: Collection, edges: Collection<(T, T)>, ) -> Collection<'static, ((T, T), usize)> { let mut subcontext = context.subgraph::(); let (var, c) = subcontext.variable(); let c = c.named("c"); let nextdists = verts .map(|x| ((x.clone(), x), 0)) .enter() .concat( c.map(|(d, (x, y))| (y, (x, d))) .join(edges.enter()) .map(|(_, ((x, d), y))| ((x, y), d + 1)), ) .group_min() .split() .named("nextdists"); var.set(nextdists.clone().map(swap)); nextdists.leave(&subcontext.finish()).collect() } } pub mod collections { use super::Binary; use crate::primitives::{Assig, RuleIndex}; use crate::program::RulesCollections; use dc2::{Collection, CreationContext}; use lazy_fields::{self, with_lazy_fields, LazyField}; use std::cell::RefCell; type PField<'a, T> = LazyField<'a, SolverCollections<'a>, T>; type RuleSizesCollection = Collection<'static, (RuleIndex, isize)>; pub struct SolverCollections<'a> { pub context: RefCell, pub all_lits: PField<'a, Collection<'static, Assig>>, pub base: PField<'a, RulesCollections>, pub active_base: PField<'a, RulesCollections>, pub rem_lits: PField<'a, Collection<'static, Assig>>, pub active: PField<'a, RulesCollections>, pub rule_sizes: PField<'a, RuleSizesCollection>, pub binary: PField<'a, Binary>, } impl<'a> SolverCollections<'a> { pub fn new() -> Self { with_lazy_fields( move |r: &mut lazy_fields::Register<'a, SolverCollections<'a>>| { SolverCollections { context: RefCell::new(CreationContext::new()), base: r.field(SolverCollections::make_base), all_lits: r.field(SolverCollections::make_all_lits), active_base: r.field(SolverCollections::make_active_base), rem_lits: r.field(SolverCollections::make_rem_lits), active: r.field(SolverCollections::make_active), rule_sizes: r.field(SolverCollections::make_rule_sizes), binary: r.field(SolverCollections::make_binary), } }, ) } fn make_all_lits(&self) -> Collection<'static, Assig> { unimplemented!() } fn make_rem_lits(&self) -> Collection<'static, Assig> { unimplemented!() } fn make_base(&self) -> RulesCollections { unimplemented!() } fn make_active_base(&self) -> RulesCollections { unimplemented!() } fn make_active(&self) -> RulesCollections { unimplemented!() } fn make_rule_sizes(&self) -> RuleSizesCollection { unimplemented!() } } } mod learnt { use crate::ops::RcRaw; use crate::primitives::{Assig, RuleIndex}; use std::cell::RefCell; use std::cmp::Ordering; use std::collections::HashMap; #[derive(Clone, PartialEq, Eq, Hash, Debug)] pub struct RefRule(RcRaw>); #[derive(Debug)] enum RuleBuilder { Leaf(RuleIndex), Node(RefRule, HashMap), } impl PartialOrd for RefRule { fn partial_cmp(&self, _other: &Self) -> Option { unimplemented!() } } impl Ord for RefRule { fn cmp(&self, _other: &Self) -> Ordering { unimplemented!() } } } mod primitives { use super::learnt::RefRule; use crate::primitives::{MicroLevel, RuleIndex}; pub type DecisionLevel = usize; #[derive(Clone, Debug, PartialEq, Eq, Hash)] pub struct AssignInfo { pub decision_level: DecisionLevel, pub cause: Cause, pub micro_level: MicroLevel, } #[derive(Clone, Debug, PartialEq, Eq, Hash)] pub enum Cause { Decision, InferredFrom(RuleIndex), Pure, BinaryChains(RefRule), } } use self::binary::Binary; use self::collections::SolverCollections; } mod tuple { pub fn fst((a, _): (A, B)) -> A { a } pub fn snd((_, b): (A, B)) -> B { b } pub fn swap((a, b): (A, B)) -> (B, A) { (b, a) } } use self::solver::collections::SolverCollections; fn main() { SolverCollections::new(); } ```

[tgnottingham]

@rustbot claim

[davidspies]

Any ideas at a workaround? I tried changing all the .split calls to .collect and that reduced the memory usage quite a bit, but after some more changes to the dc2 package (https://github.com/davidspies/dc2/commit/d73daf710bf02da48aeb8f2c308575e676e58d5c) it's still using more than 32GiB memory

[tgnottingham]

I haven't been able to fully investigate, so I can't help much yet. But the issue is partly that the compiler is struggling with generic functions whose type parameters are deeply nested. The MCVE results in types whose string representations are extremely long (like FlatMap<FlatMap<Receiver<FlatMap<Concat<Receiver<FlatMap<Join<Receiver<...>>>>>>>>). One is 19 million characters.

At one point, I believe these might have been rejected by the default type length limit, but there is a bug preventing that.

These types also cause the compiler to generate enormous symbol names, which accounts for the memory increase in the partition_and_assert_distinct_symbols pass. They're likely a problem during code generation as well.

[davidspies]

Thanks,

How do I see how long the generated types are?

I want to try various changes and see how that affects things

[tgnottingham]

You can use -Z print-mono-items=lazy. The output will be a couple of GB.

[tgnottingham]

@davidspies, a partial solution is to use -Z symbol-manging-version=v0. That enables the newer symbol mangling scheme, which generates smaller symbols in cases where there are repeated components within it. With this, the MCVE compiles in 1.3 GB in debug mode.

However, compiling in release mode (#82445) still requires 14 GB. You can lower this substantially by disabling all LTO, for example by putting this in Cargo.toml:

[profile.release]
lto = "off"

With LTO off in release mode, it compiles using ~875MB.

But whether in debug or release mode, it still takes a longer time to compile than one would hope. Much of the time appears to be spent in the call to generate the optimized MIR. Still need to investigate that and the LTO memory usage.

By the way, there are several related issues that this may turn out to be a duplicate of, for example, #54540.

[davidspies]

thanks so much; I'll try these things

[davidspies]

I'm using vscode with the rust extension. Is there some way to get rls to build with the -Z option? Can I modify the Cargo.toml for that?

[jyn514]

I would not recommend using RLS for anything complicated. matklad.rust-analyzer works pretty well, there's some setting to change the arguments to rustc I think.

[jyn514]

Alternatively you could put this in .cargo/config, which should have cargo pick it up automatically.

[davidspies]

I tried creating a .cargo/config.toml file with:

[target.nightly-x86_64-unknown-linux-gnu]
rustflags = "-Z symbol-manging-version=v0"

note I kept your typo sayng "manging" rather than "mangling" and it didn't complain. I think it's ignoring the settings in my config.toml. I don't know whether that's because I have the wrong target name or because I'm not supposed to have the -Z. I can't find any example toml files that do it correctly.

[jyn514]

@davidspies remove nightly:

[target.x86_64-unknown-linux-gnu]
rustflags = "-Z symbol-mangling-version=v0"

[tgnottingham]

@rustbot release-assignment

Sorry, I haven't had time to work on this.

[davidspies]

@tgnottingham Is there a way to ask someone else to take it on? Also, how do I get rid of the "needs-mcve" tag?

[jyn514]

@davidspies do you mean https://github.com/rust-lang/rust/issues/82406#issuecomment-783664175 for the MCVE? I think it might still be nice to get it smaller (at least no dependencies).