Segfault while building the Rust toolchain with a toolchain that uses mimalloc

gnzlbg commented 5 years ago

So I'm trying to build a Rust toolchain with mimalloc that works here: https://github.com/rust-lang/rust/pull/62340 , such that we can use the Rust toolchain compiler benchmark suite to test whether it would be worth it switching to mimalloc from jemalloc.

I think I've managed so far to link mimalloc properly, and built a toolchain that's statically linked with it and uses it. When that toolchain is used to build another toolchain, after building a chunk of it, it segfaults with: signal: 11, SIGSEGV: invalid memory reference. Target is x86_64-unknown-linux-gnu , and I'm using the master branch of mimalloc, via the mimalloc-sys bindings, which are here: https://github.com/gnzlbg/mimallocator/tree/master/mimalloc-sys

I'm building mimalloc following the steps from this script: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/build.rs

gnzlbg commented 4 years ago

@daanx what you can do is add the --on-fail=bash flag to the x.py invocation. Once something fails, it will create an interactive shell with the environment properly set up. In that shell you can then spawn a debugger to run the command that failed.

gnzlbg commented 4 years ago

@daanx if --on-fail=bash does not work, you can use --on-fail=env to print the environment that's used by x.py. And then set it up yourself before trying to reproduce by executing the command that failed.

gnzlbg commented 4 years ago

Has there been any progress on this?

daanx commented 4 years ago

I would like to try this again with the latest dev (or dev-abandon) mimalloc branches. However, it would be great if I can control the mimalloc sources; as I understand it is now pulled from https://github.com/gnzlbg/mimallocator. What is the best way to change this so I can change the source or (cmake) compilation flags? Can it point to a local folder?

ArniDagur commented 4 years ago

Try this patch:

diff --git a/src/rustc/Cargo.toml b/src/rustc/Cargo.toml
index 825e3655ac..7a4f81ce59 100644
--- a/src/rustc/Cargo.toml
+++ b/src/rustc/Cargo.toml
@@ -22,7 +22,7 @@ rustc_codegen_ssa = { path = "../librustc_codegen_ssa" }
 #features = ['unprefixed_malloc_on_supported_platforms']

 [dependencies.mimalloc-sys]
-version = '=0.1.6-dev'
+path="../../../mimallocator/mimalloc-sys"
 optional = true

 [features]

After something like:

git clone git@github.com:gnzlbg/rust.git
cd rust
git checkout mimalloc
cd ..
git clone git@github.com:gnzlbg/mimallocator.git
cd mimallocator/mimalloc-sys
rmdir mimalloc
git clone git@github.com:microsoft/mimalloc.git
cd mimalloc
git checkout dev

Remember to enable jemalloc in config.toml or using the configure script.

ArniDagur commented 4 years ago

For what it's worth, I got Rust to compile with only these messages from mimalloc:

   Compiling rustc_resolve v0.0.0 (/sdb/src/rust/src/librustc_resolve)
   Compiling rustc_privacy v0.0.0 (/sdb/src/rust/src/librustc_privacy)
   Compiling rustc_plugin v0.0.0 (/sdb/src/rust/src/librustc_plugin/deprecated)
   Compiling rustc_codegen_ssa v0.0.0 (/sdb/src/rust/src/librustc_codegen_ssa)
   Compiling rustc_save_analysis v0.0.0 (/sdb/src/rust/src/librustc_save_analysis)
mimalloc: warning: possibly trying to free a pointer that does not point to a valid heap region: 0x0x40030000100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40030000100 was valid after all)
mimalloc: warning: possibly trying to free a pointer that does not point to a valid heap region: 0x0x40040000100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40040000100 was valid after all)
mimalloc: warning: possibly trying to free a pointer that does not point to a valid heap region: 0x0x40074400100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40074400100 was valid after all)
   Compiling rustc-main v0.0.0 (/sdb/src/rust/src/rustc)
    Finished release [optimized] target(s) in 9m 48s
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage1 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
   Compiling cc v1.0.35

daanx commented 4 years ago

Thank you @ArniDagur, @gnzlbg: with the above instructions I successfully built using the latest dev-abandon branch (on its way to become version 2.0 with dev to 1.3). However, when enabling the secure build it still fails! The error was:

mimalloc: warning: mprotect error: start: 0x0x4ab9e75f000, csize: 0x1000, err: 12

(actually, the real error occurred later with a panic message a thread could not start due to resource exhaustion)

Some careful reading shows errno 12 is ENOMEM, and the man page says that that might be caused by creating too many mappings. Well, of course, full secure mode creates guard pages between every mimalloc "page" thus creating many mappings, and I guessed the biggest compiled sources may exceed the default (64k mappings) so I increased it to 256k:

sudo sysctl -w vm.max_map_count=262144

and that was it! After that it built in secure mode with full debug checking. :-) Great. I will add a better warning message and add documentation; and perhaps a secure mode that only uses guard pages around full segments instead of every mimalloc page.

Pfff, glad this got sorted out -- would love to see some benchmarks with rust.