Segfault while building the Rust toolchain with a toolchain that uses mimalloc

[gnzlbg]

So I'm trying to build a Rust toolchain with mimalloc that works here: https://github.com/rust-lang/rust/pull/62340 , such that we can use the Rust toolchain compiler benchmark suite to test whether it would be worth it switching to mimalloc from jemalloc.

I think I've managed so far to link mimalloc properly, and built a toolchain that's statically linked with it and uses it. When that toolchain is used to build another toolchain, after building a chunk of it, it segfaults with: signal: 11, SIGSEGV: invalid memory reference. Target is x86_64-unknown-linux-gnu , and I'm using the master branch of mimalloc, via the mimalloc-sys bindings, which are here: https://github.com/gnzlbg/mimallocator/tree/master/mimalloc-sys

I'm building mimalloc following the steps from this script: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/build.rs

[daanx]

Very nice you are working on integrating this with Rust -- let me know of issues or performance. With regard to the crash, when integrating with a language, it is best to:

• if the runtime uses plain malloc/free, then first use LD_PRELOAD to see if those programs still work. This lets you know if the bug is in mimalloc or in the integration.
• for a language runtime, it is best to link statically, and have the runtime consistently use the mi_ prefixed API and never mix up pointers gotten through some other API. However, that means special care must be taken that pointers obtained through some C library do not get freed with the new Rust free (which uses mi_free). In particular, watch for uses of strdup, strndup, and realpath ! and use the mi_ prefixed versions instead. Same for posix_align etc. (this will get easier once I get to an update for issue #75 )

Do you have a backtrace of the segfault? Was it using the debug version of mimalloc with -DMI_CHECK_FULL=ON enabled?

I will try help out and try to look at the Rust thread tomorrow.

edit: I looked a little and saw rust already works with jemalloc so the above issues should already be handled well. Don't use MI_OVERRIDE and statically link seems the best approach (which you are already doing).

[daanx]

Just a note, ensure the aligment and size arguments are in the right order for the aligned allocation functions in:

        static _F2: unsafe extern fn(*mut *mut c_void, usize, usize) -> c_int =
            mimalloc_sys::posix_memalign;
        static _F3: unsafe extern fn(usize, usize) -> *mut c_void =
            mimalloc_sys::aligned_alloc;

since mimalloc lets alignment always come after the size but posix_memalign and memalign do it the other way around!

[gnzlbg]

let me know of issues or performance.

First step is to get it to work correctly, but that's the next step :)

if the runtime uses plain malloc/free, then first use LD_PRELOAD to see if those programs still work. This lets you know if the bug is in mimalloc or in the integration.

So in the mimalloc-sys crate, I do two things:

• check that mimalloc is actually used: with the MIOVERRIDE setting, mimalloc replaces the standard symbols, but still provides the `misymbols. This was great to be able to test that the library was properly linked, e.g., by callingmi_malloc+freeandmalloc+mi_free, and making sure that works. (When I wrote the tests the first time, they failed, because I didn't hadMI_INTERPOSE` enabled on macos, so hurd the system allocator was trying to free memory from mimalloc and vice-versa, which crashed).

• c-ffi-test: for all functions, types, etc. exposed by the Rust bindings to a C API, tests are generated that check that the size, alignment, field offsets, etc. of types are the same, that the addresses of the rust function wrapper and the C functions is the same, that the functions take the same arguments, return types, etc. Essentially, that the ABI of the wrapper matches that of the library being wrapped. This is done by generating C code and Rust code, to compute these things at each side of the FFI, and then collecting the results and compare them.

if the runtime uses plain malloc/free, then first use LD_PRELOAD to see if those programs still work. This lets you know if the bug is in mimalloc or in the integration.

The Rust compiler uses "plain" malloc/free (the C and POSIX apis), but right now jemalloc is statically linked into the binary, overriding these symbols. I know LD_PRELOAD works if the malloc library is linked dynamically, but I'm not sure if it works in this situation.

By default, Rust binaries use the system allocator. I've tried DYLD_LIBRARY_INSERT=.. with mimalloc on some small binary and that just worked (I saw the statistics output), but the Rust compiler is a beast in code size, and number of tests and benchmarks, and other linked libraries, so it

for a language runtime, it is best to link statically, and have the runtime consistently use the mi_ prefixed API and never mix up pointers gotten through some other API. However, that means special care must be taken that pointers obtained through some C library do not get freed with the new Rust free (which uses mifree). In particular, watch for uses of strdup, strndup, and realpath ! and use the mi prefixed versions instead. Same for posix_align etc. (this will get easier once I get to an update for issue #75 )

So I'm statically linking mimalloc into the binary, and hopefully correctly overriding all symbols. That's what we do for jemalloc and it works. So I would hope that this would override strdup and friends as well.

One thing we are doing is compiling LLVM and somehow linking it into rustc. We don't do anything special for jemalloc here, so I don't think we need to do anything special for mimalloc either.

Just a note, ensure the aligment and size arguments are in the right order for the aligned allocation functions in:

Those are wrapped here: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/src/lib.rs#L49

We just provide the C and POSIX API. I suppose the ones mimalloc defines have the arguments in the standard orders for these. From looking at the code here, they appear to match: https://github.com/microsoft/mimalloc/blob/master/src/alloc-override.c#L138

That is, we are only using the overriden functions from mimalloc, we don't use any of the mi_... APIs directly in the compiler (yet).

Do you have a backtrace of the segfault? Was it using the debug version of mimalloc with -DMI_CHECK_FULL=ON enabled?

AFAICT it was using a release build, but I will enable full checks and try again.

[daanx]

Those are wrapped here: https://github.com/gnzlbg/mimallocator/blob/master/mimalloc-sys/src/lib.rs#L49

We just provide the C and POSIX API. I suppose the ones mimalloc defines have the arguments in the standard orders for these. From looking at the code here, they appear to match: https://github.com/microsoft/mimalloc/blob/master/src/alloc-override.c#L138

Mmm, I mean the mi_malloc_aligned and friends don't match with posix_memalign: the arguments are swapped:

int posix_memalign(void** p, size_t alignment, size_t size);
void* memalign(size_t alignment, size_t size);

versus

void* mi_malloc_aligned(size_t size, size_t alignment);

Be very careful when binding to Rust that it matches what Rust defined.

[gnzlbg]

Mmm, I mean the mi_malloc_aligned and friends don't match with posix_memalign:

Ah ok. Yes, we don't use mi_... and friends, only malloc/posix_memalign, etc. and the APIs of those that we use appear correct.

[daanx]

I understand you want to link statically with mimalloc and override the standard malloc/free. If so, that will still be tricky general. Static + override:

• works on Linux/BSD with weak symbols -- just link carefully to ensure mimalloc comes first
• should work on macOS -- but currently there are issues (#50) but I am confident it will be fixed in the future as other allocators make this work too. More testing is needed here and I need to study how jemalloc does this, tcmalloc uses MacOS zones which I am currently testing with.
• does not work on Windows with msvc! (needs dynamic loading)

Static but not overriding is the safest option but you need to control the runtime system enough to ensure no pointers are mixed -- but that may well be the case for rust.

For now, I would at least try your current approach on Linux first to ensure it all works before moving on to other platforms.

[gnzlbg]

I forgot to mention that, to be able to run Rust compiler benchmarks, we only need to be able to link mimalloc on Linux.

Static but not overriding is the safest option but you need to control the runtime system enough to ensure no pointers are mixed -- but that may well be the case for rust.

I don't think this is an option. We link against LLVM, which uses the malloc APIs to allocate memory. I can't really assume that there aren't any places in the code where memory from one is passed to the other, which afterwards deallocates it. I hope this isn't the case, but making sure that it isn't would be a lot of work at this point (EDIT: the code interfacing with LLVM is compartmentalized in a single place, so while doable, it is still a lot of work to do this).

For now, I would at least try your current approach on Linux first to ensure it all works before moving on to other platforms.

That's the plan. Maybe I wasn't clear. Our current goal isn't to switch from jemalloc to mimalloc in the Rust toolchain, but rather to gather data about the performance of the Rust toolchain when mimalloc is its allocator. Performance is just one of the many data-points that would need to be considered in a discussion about doing a switch, but that discussion hasn't started yet. This is purely exploratory work.

If you want to try my PR branch on Linux, the only thing you have to do is clone it, and execute ./x.py build --stage 1. It should work out-of-the box.

[daanx]

Thanks for explaining :-) I thought that you were working on macOS somehow. Anyway, in that case there should not be segfaults so let's see if we can track down the cause. If can provide me with a test case, or a debug backtrace from the faulty build that will help. If I try your branch, will it cause the error for me investigate?

[gnzlbg]

If I try your branch, will it cause the error for me investigate?

Yes. Briefly, the Rust toolchain bootstraps itself. So the first "stage" is stage0, and here, an older compiler is used to compile the toolchain, producing a stage0 compiler. This stage0 compiler that gets produced here is linked statically with mimalloc in my PR branch. Then, in stage1, this compiler is used to compile the toolchain again, producing a stage1 compiler. This is where the compiler using mimalloc gets exercised, and where the error happens.

The ./x.py script handles everything for you. To build a stage1 compiler, one can write ./x.py build --stage 1. Unless you want to modify the stage0 compiler, e.g., to link mimalloc differently, you probably don't want to re-build stage0 all the time. You can prevent that by doing ./x.py build --stage 1 --keep-stage 0. When compilation fails, you get an error message, providing the CLI invocation that failed to execute. So you can just copy-paste it into your terminal, and re-try, use gdb, etc. Please let me know if you have any questions or need any help. If you start doing this, and happen to have Discord or Zulip installed, you can join the Rust channels there (search for Rust discord and Rust zulip for the links). There are compiler channels in both, where you can ask any questions.

[daanx]

This is still on my todo list -- let me know of further issues but I will get to it soonish :-)

[gnzlbg]

@daanx let me know if you need help setting anything up or feel free to drop by Rust's discord or Zulip chat rooms.

[ArniDagur]

Building rustc seems to work fine for me, am I doing something wrong?

git clone git@github.com:gnzlbg/rust.git rust-mimalloc
cd rust-mimalloc
git checkout mimalloc
./x.py build

Results in:

Build completed successfully in 0:37:10

If I do ./x.py build --stage 1 --keep-stage 0, I get the same.

[daanx]

The issue is fairly old and in the meantime various bugs were addressed. Perhaps it is just working now? :-) If so, please confirm as it would be nice if Rust is able to easily use mimalloc as an allocator. I read somewhere that Rust now uses no default allocator but instead let's the user opt-in to another allocator -- it would be great if mimalloc can be one of those options.

[mati865]

Steps from https://github.com/microsoft/mimalloc/issues/77#issuecomment-530619403 give you 2 months old Rust and Mimalloc. There are no fixes included so it shouldn't work meaning something went wrong.

I read somewhere that Rust now uses no default allocator but instead let's the user opt-in to another

Rust compiler uses jemalloc but users can choose which allocator they will use in their binaries (defaults to system allocator).

[gnzlbg]

@mati865 actually, the latest commit in that PR was never "try"ed (some other PR landed before that that made it unmergeable), so... maybe it worked ? I'll rebase and we should schedule a try-build.

[gnzlbg]

So I rebased on top of master, and re-run all tests, and we are still failing with a SIGSEV: https://dev.azure.com/rust-lang/e71b0ddf-dd27-435a-873c-e30f86eea377/_build/results?buildId=7960

[gnzlbg]

@ArniDagur which platform are you testing on ? Those build jobs run on x86_64-linux. It might also be that there are some differences between how those jobs are build there and your machine.

[daanx]

@gnzlbg: thanks for trying again!. I am still very interested to fix this and understand what is happening. Can you try with the latest dev branch too? And if that still fails, perhaps build with -DMI_CHECK_FULL=ON so we can see if any assertions fail. (Perhaps too much work, no worries if you don't get to it)

[ArniDagur]

Here's information about my system:

• OS: Gentoo Linux
• Output of uname -a: Linux arni-pc 5.2.3-ck-x86_64 #1 SMP Wed Sep 11 19:29:52 GMT 2019 x86_64 Intel(R) Core(TM) i7-4790 CPU @ 3.60GHz GenuineIntel GNU/Linux
• Compiler: gcc (Gentoo 9.2.0 p1) 9.2.0
• Linker: GNU ld (Gentoo 2.32 p2) 2.32.0
• Compiler environment variables: RUSTFLAGS=-C target-cpu=native

[mati865]

Builds ran through Docker containers have various config options tweaked: https://github.com/rust-lang/rust/blob/c43d03a19f326f4a323569328cc501e86eb6d22e/src/ci/run.sh DEPLOY=1 is defined BTW.

[gnzlbg]

Can you try with the latest dev branch too? And if that still fails, perhaps build with -DMI_CHECK_FULL=ON so we can see if any assertions fail. (Perhaps too much work, no worries if you don't get to it)

The results are in here: https://dev.azure.com/rust-lang/rust/_build/results?buildId=7973&view=logs&jobId=d0d954b5-f111-5dc4-4d76-03b6c9d0cf7e&taskId=1094049d-8071-5d7a-acdc-b0909289cb44&lineStart=14107&lineEnd=14107&colStart=1&colEnd=75

An assertion triggers:

mimalloc: assertion failed: at "mimalloc/src/page.c":324, _mi_page_abandon
  assertion: "_mi_ptr_page(block) != page"

[daanx]

Ah very interesting -- that assertion should not trigger! Can you try if this happens with the latest dev branch too? I made various improvements to atomic handling which may fix it. I will try to repro on my local linux box if I can find some time. Thanks again for your work -- hope I can figure out what is causing this.

[gnzlbg]

@daanx that test was done with the latest dev branch that was released at the time of the test (~3 hours ago), but the last commit to the dev branch is from 17 hours ago, so those tests should have used the very latest dev branch. If you update it feel free to let me know and I'll re-run the tests.

[daanx]

Ah I see -- but it says line 324 in the assertion which corresponds to the master branch, in dev it is line 343. Maybe a wrong checkout?

[gnzlbg]

Weird, I noticed that to. I'll investigate.

[gnzlbg]

You were right. I was being bitten by semver, the "0.1.6-dev" version was being resolved as "0.1.6". I had to specify that I really really wanted that one with "=0.1.6-dev". I've restarted the build.

[goffrie]

From the latest run...

2019-09-12T23:14:31.9142450Z mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x413f1c02000
2019-09-12T23:14:31.9142691Z (this may still be a valid very large allocation (over 64MiB))
2019-09-12T23:14:31.9142806Z mimalloc: warning: (yes, the previous pointer 0x0x413f1c02000 was valid after all)
2019-09-12T23:14:31.9292779Z mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x41426002000
2019-09-12T23:14:31.9292911Z (this may still be a valid very large allocation (over 64MiB))
2019-09-12T23:14:31.9294975Z mimalloc: warning: (yes, the previous pointer 0x0x41426002000 was valid after all)
2019-09-12T23:18:21.3635930Z [RUSTC-TIMING] rustc test:false 821.997
2019-09-12T23:18:21.3636999Z rustc exited with signal: 11

[gnzlbg]

@goffrie showed the error, here is the link.

[ArniDagur]

@gnzlbg I managed to reproduce the crash: I just needed to put "--set rust.jemalloc" in the ./configure arguments. This script:

#!/bin/bash

# Clean up from previous build
rm -rf build

# Configuration which is always enabled
export RUST_RELEASE_CHANNEL=nightly
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --enable-sccache"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --disable-manage-submodules"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --enable-locked-deps"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --enable-cargo-native-static"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --set rust.codegen-units-std=1"

# Deploy configuration
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --release-channel=$RUST_RELEASE_CHANNEL"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --enable-llvm-static-stdcpp"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --set rust.remap-debuginfo"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --debuginfo-level-std=1"

RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --enable-llvm-assertions"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --set rust.verify-llvm-ir"

RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --enable-debug-assertions"
RUST_CONFIGURE_ARGS="$RUST_CONFIGURE_ARGS --set rust.jemalloc"

# Configure and make
./configure $RUST_CONFIGURE_ARGS
make -j $(nproc)

Produces this:

mimalloc: warning: mprotect error: start: 0x0x42a0beff000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf0f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf1f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf2f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf3f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf4f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf5f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf6f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf7f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf8f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42a0bf9f000, csize: 0x1000, err: -1
error: Could not compile `rustc`.

Caused by:
  process didn't exit successfully: `/home/arni/src/rust-mimalloc/build/bootstrap/debug/rustc --edition=2018 --crate-name rustc src/librustc/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=2 -C metadata=f648942e8bb9ba69 -C extra-filename=-f648942e8bb9ba69 --out-dir /home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps --extern arena=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libarena-ebc900195d83aa4d.rmeta --extern backtrace=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbacktrace-f722349a2b3cfdff.rmeta --extern bitflags=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbitflags-58e0dbae211d255d.rmeta --extern byteorder=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbyteorder-c31e835416d3ff9d.rmeta --extern chalk_engine=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libchalk_engine-8c24322dc01837b3.rmeta --extern fmt_macros=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libfmt_macros-8515eb6d1fcab541.rmeta --extern graphviz=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libgraphviz-70da08ae09e4c746.rmeta --extern jobserver=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libjobserver-f8e516e6a924bac0.rmeta --extern lazy_static=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/liblazy_static-a1357debc52c144b.rmeta --extern log=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/liblog-09b8d7c33ca043bf.rmeta --extern measureme=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libmeasureme-04821d071e81439f.rmeta --extern num_cpus=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libnum_cpus-07a7b672ea534d8c.rmeta --extern parking_lot=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libparking_lot-0e0dd756a727d78f.rmeta --extern polonius_engine=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libpolonius_engine-459cd13d7ed8d19e.rmeta --extern rustc_rayon=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_rayon-74e7aa8a5023fbd2.rmeta --extern rustc_rayon_core=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_rayon_core-114a2a36aae6da4c.rmeta --extern rustc_apfloat=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_apfloat-fac7e66c915effcc.rmeta --extern rustc_data_structures=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_data_structures-5b591454e16cc41e.rmeta --extern errors=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_errors-1640a4e8e73821c6.rmeta --extern rustc_fs_util=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_fs_util-055a87fca5033c2d.rmeta --extern rustc_macros=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps/librustc_macros-09874915f7420f3f.so --extern rustc_target=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_target-98feb11063278230.rmeta --extern scoped_tls=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libscoped_tls-bbe59bc685d5bfcc.rmeta --extern rustc_serialize=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-eb2cf79e1d097189.rmeta --extern smallvec=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsmallvec-29b1fb24aa27bec5.rmeta --extern syntax=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax-f69f9c56af06e538.rmeta --extern syntax_pos=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax_pos-cb972f90722f72c8.rmeta -C target-cpu=native -Zbinary-dep-depinfo --cfg always_verify_llvm_ir -L native=/home/arni/src/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/backtrace-sys-00a11e73bbe3a3bf/out` (signal: 11, SIGSEGV: invalid memory reference)
warning: build failed, waiting for other jobs to finish...
mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x420fc402000
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x420fc402000 was valid after all)
^C  Building [====================================================>  ] 155/158: rustc_mir
Build completed unsuccessfully in 0:23:26
make: *** [Makefile:18: all] Error 1

[daanx]

Very interesting!

1. As I understand, the setting --set rust.jemalloc causes the crash right? Perhaps that means that parts are compiled with mimalloc while other parts link with jemalloc and a mimalloc'd pointer is freed by jemalloc (or vice versa)?
2. The mprotect error is quite strange and should not happen with the mimalloc default settings as it only uses mprotect to "commit" on demand, or when in secure mode. I wonder what is going on. It is also exactly 1 4096 bytes every time -- perhaps it happens with thread allocations? I will investigate if I can.

Thanks for trying to make this work!

[gnzlbg]

I just needed to put "--set rust.jemalloc" in the ./configure arguments.

Yeah, this might have been the reason this was not reproducing. IIRC rustc builds locally with the system allocator, and only on CI and for shipping binaries do we use a different allocator (jemalloc right now).

Perhaps that means that parts are compiled with mimalloc while other parts link with jemalloc and a mimalloc'd pointer is freed by jemalloc (or vice versa)?

I checked and this isn't happening. The reason the option is still called jemalloc is because I was just lazy and wanted to avoid updating all CI jobs, the build options, the docs, etc. so I just left jemalloc everywhere, except in the parts where the library is actually linked, which got changed to use mimalloc.

The mprotect error is quite strange and should not happen with the mimalloc default settings as it only uses mprotect to "commit" on demand, or when in secure mode.

I'm building mimalloc with override, secure mode, and check full.

[daanx]

Ah thanks for the clarification. Can we try to build without secure mode (but with all debug checks)? That should hopefully remove the mprotect warnings.

Still, strange the crash only happens with jemalloc set to true, but not without it? What can it be?

[gnzlbg]

Can we try to build without secure mode (but with all debug checks)?

Opened a PR here: https://github.com/rust-lang/rust/pull/64495#issuecomment-531593562 I'll report when the CI results become available.

Still, strange the crash only happens with jemalloc set to true, but not without it?

If the jemalloc flag is set to true, the jemalloc allocator is statically linked to the Rust compiler - I've changed the implementation of the flag to statically link mimalloc instead.

If the jemalloc flag is set to false, the system allocator is dynamically linked instead (e.g. glibc on linux).

What can it be?

I think that from more to less plausible, eIther there is a bug in how we link mimalloc, or there is a bug in mimalloc, or there is a bug in rustc that only surfaces with mimalloc, but not all other allocators (rustc works on all platforms on which firefox is built, and those are too many different memory allocators).

I think that the brittlest spot is probably how we build mimalloc for usage in Rust. We prefer not to use cmake to avoid requiring it as a dependency, so we just use the system C compiler to build the files manually, passing it the #defines manually, and then link them statically with the Rust compiler, just like how we do for jemalloc. I've tried to be careful with porting the defines, C flags used by CMake, etc. but it is likely that I've made a mistake there. Maybe we are also linking it incorrectly? I mean, it is just a static C library, and we statically link it, but maybe mimalloc requires some extra options?

I think it is also plausible that there is a bug in mimalloc. The Rust compiler is self-hosting, that is, it bootstraps itself. This process works as follows. We use an older already-existing compiler (that doesn't use mimalloc) to compile the sources - this produces the stage0 compiler which corresponds to the code in the repository and it is therefore linked against mimalloc (we know it is linked, because mimalloc produces a lot of debug output with some settings, and we see that). Then this stage0 compiler is used to recompile all sources again, producing a stage 1 compiler. This is the step that fails. The weird thing is that this stage0 compiler which uses mimalloc is able to compile ~100 rust translation units/libraries before actually starting to fail (the rust compiler is like ~160 translation units). The rust compiler is huge, and compiling 100 crates exercises a lot of code, not only in Rust but also in LLVM, which is also huge. So I'd expected that if there was something off with how we linking or configure mimalloc, that this stage0 compiler would have actually failed much earlier.

[daanx]

Thanks for the great explanation. I'll ponder on it :-)

[gnzlbg]

Build failed: https://dev.azure.com/rust-lang/rust/_build/results?buildId=8007&view=logs&jobId=d0d954b5-f111-5dc4-4d76-03b6c9d0cf7e&taskId=1094049d-8071-5d7a-acdc-b0909289cb44&lineStart=13850&lineEnd=13851&colStart=1&colEnd=1

2019-09-15T21:24:31.2652266Z warning: build failed, waiting for other jobs to finish...
2019-09-15T21:27:09.3440987Z mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x40040000100
2019-09-15T21:27:09.3499463Z (this may still be a valid very large allocation (over 64MiB))
2019-09-15T21:27:09.3499642Z mimalloc: warning: (yes, the previous pointer 0x0x40040000100 was valid after all)
2019-09-15T21:27:09.3531688Z mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x40074400100
2019-09-15T21:27:09.3536360Z (this may still be a valid very large allocation (over 64MiB))
2019-09-15T21:27:09.3540136Z mimalloc: warning: (yes, the previous pointer 0x0x40074400100 was valid after all)
2019-09-15T21:30:52.7172375Z [RUSTC-TIMING] rustc test:false 828.046
2019-09-15T21:30:52.7175436Z rustc exited with signal: 11
2019-09-15T21:30:52.7235712Z error: Could not compile `rustc`.

[daanx]

I did:

git clone https://github.com/gnzlbg/rust.git rust-mimalloc
cd rust-mimalloc
git checkout mimalloc
./x.py build

and it build without errors. Then I used @ArniDagur script above with the "--set rustc.jemalloc" and everything succeeded? No errors.

I am using Ubuntu 18.04.3 LTS.

Is there an easy way to check the results or run just the past that fails in the CI? Best, Daan

ps. part of the output is:

...
  Compiling rustc_plugin v0.0.0 (/home/daan/dev/rust-mimalloc/src/librustc_plugin/deprecated)
mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x40030000100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40030000100 was valid after all)
mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x40040000100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40040000100 was valid after all)
mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x40074400100
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x40074400100 was valid after all)
   Compiling rustc-main v0.0.0 (/home/daan/dev/rust-mimalloc/src/rustc)
    Finished release [optimized] target(s) in 12m 15s
Copying stage1 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Building stage1 codegen artifacts (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu, llvm)
   Compiling cc v1.0.35
   Compiling build_helper v0.1.0 (/home/daan/dev/rust-mimalloc/src/build_helper)
   Compiling rustc_codegen_llvm v0.0.0 (/home/daan/dev/rust-mimalloc/src/librustc_codegen_llvm)
   Compiling rustc_llvm v0.0.0 (/home/daan/dev/rust-mimalloc/src/librustc_llvm)
    Finished release [optimized] target(s) in 1m 05s
Assembling stage2 compiler (x86_64-unknown-linux-gnu)
Uplifting stage1 std (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 std from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
Copying stage2 rustc from stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
... 

[gnzlbg]

I did: [...] and it build without errors.

Yep, that's not using mimalloc but your system allocator.

Then I used @ArniDagur script above with the "--set rustc.jemalloc" and everything succeeded? No errors.

Hmm. No idea. @ArniDagur could you retry your script with the latest branch ? (you are going to need to re-pull since I had to force push =/)

Is there an easy way to check the results or run just the past that fails in the CI?

@mati865 do you know how to make the CI script only build up to stage1 ? That should cut ~1/3 of @daanx workflow. @daanx maybe you can try also passing the script the flag --stage 1.

---

@daanx if you want to retry with the secure build, you just need to locally undo the last commit (https://github.com/rust-lang/rust/pull/64495/commits/111e3f8be81b935507baf789882a0cf668afcbea).

[mati865]

EDIT: I could not make latest checkout to crash even in secure mode on Arch Linux:

   Compiling rustc_plugin v0.0.0 (/home/mateusz/Projects/rust/rust/src/librustc_plugin/deprecated)
mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x43f3a802000
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x43f3a802000 was valid after all)
mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x42d8b402000
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x42d8b402000 was valid after all)
mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x42dbf802000
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x42dbf802000 was valid after all)
mimalloc: warning: mprotect error: start: 0x0x42de64bf000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42de64cf000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42de64df000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42de64ef000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42de64ff000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42de650f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x42e1f001000, csize: 0x1000, err: -1
   Compiling rustc-main v0.0.0 (/home/mateusz/Projects/rust/rust/src/rustc)

---

@gnzlbg this crash requires stage 0 and half of stage 1 so yeah ./x.py build --stage 1 should do the job.

Then I used @ArniDagur script above with the "--set rustc.jemalloc" and everything succeeded? No errors.

Latest commit disables secure mode making it work despite the warnings, you have to revert it to get the crash.

---

Script mentioned above is a bit overkill since it removes whole build/ directory causing LLVM to be rebuilt every time.

To get the crash you should revert commit disabling secure mode and just run:

./configure --set rust.jemalloc
./x.py --stage 1

Sometimes after modifying code you can get unrelated build errors. Run ./x.py clean to remove Rust objects while keeping LLVM and bootstrap compiler.

[gnzlbg]

Latest commit disables secure mode making it work despite the warnings, you have to revert it to get the crash.

@mati865 while the last commit does do that, in rust-lang/rust CI, the build still fails at some point after the warnings with a segfault, which @daanx is not able to reproduce. I don't understand why this happens. Maybe the resources of the VMs used there play a role here ? (e.g. maybe a higher memory pressure is required to trigger this?)

[daanx]

Thanks everyone for helping to isolate the problem. I hope I can repro at some point and get a trace. I am focusing on the non-secure build as that still crashed in the CI; Last time I built with make -j1 and will try now with make -j4 -- perhaps it is a concurrency bug in mimalloc that only happens with clang (and not with gcc/msvc as I usually test with those)?

[mati865]

./x.py uses all available threads, so it was 16 in my case. Clang could be worth trying, although it will use system allocator.

@daanx have you applied all the options from https://github.com/microsoft/mimalloc/issues/77#issuecomment-531462160 or only --set rust.jemalloc?

[daanx]

@mati865: I used the full script in comment #77. Maybe using sccache is not good? I also set export MIMALLOC_VERBOSE=1 so I can observe mimalloc being used. Btw. where is the mimalloc source in the tree -- I want to turn off the threadinit/done messages as there are so many.

[mati865]

Maybe it has something to do with the ancient CentoOS 5 that is used by Rust dist builders?

Btw. where is the mimalloc source in the tree

It's complicated because it's external crate: https://github.com/gnzlbg/mimallocator/tree/master/mimalloc-sys Moving it into Rust repo for testing seems like a good idea but I'm out of time.

[daanx]

Update: using make -j4 succeeds as well. I will now roll back one commit and use the mimalloc secure build and try to repro with that. I am wondering what is different on the CI machines? I remember the mprotect warnings too which are strange; I

daan@daan-HP-Z4-G4:~$ clang --version
clang version 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
daan@daan-HP-Z4-G4:~$ gcc --version
gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

daan@daan-HP-Z4-G4:~$ linuxinfo
Linux daan-HP-Z4-G4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019
Eight Intel Unknown 3829MHz processors, 57600.00 total bogomips, 15764M RAM
System library 2.27.0

[daanx]

Moving it into Rust repo for testing seems like a good idea but I'm out of time. No worries -- thanks for helping out!

[gnzlbg]

Mimalloc is pulled in via an external library, that includes the source code, a script for how to compile it, and Rust C FFI bindings so that it can be called from Rust.

That code is imported from the src/rustc/Cargo.toml file, which points to mimalloc-sys 0.1.6-dev, which lives in the dev branch of this repo: https://github.com/gnzlbg/mimallocator in the mimalloc-sys sub-directory.

[mati865]

I am wondering what is different on the CI machines?

If you have a lot of time to spend there is possibility to run exactly the same environment as Rust's CI (but it takes around twice as long). You need Linux with working Docker and then just use this in the repo: DEPLOY=1 ./src/ci/docker/run.sh dist-x86_64-linux.

[daanx]

When using the secure build, I can repro. The command fails with:

> rm -rf build
> ./configure --set rustc.jemalloc=1
> make -j4
...
   Compiling rustc_save_analysis v0.0.0 (/home/daan/dev/rust-mimalloc/src/librustc_save_analysis)
   Compiling rustc_plugin v0.0.0 (/home/daan/dev/rust-mimalloc/src/librustc_plugin/deprecated)
mimalloc: warning: possibly trying to mi_free a pointer that does not point to a valid heap region: 0x0x4389b002000
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: (yes, the previous pointer 0x0x4389b002000 was valid after all)
mimalloc: warning: mprotect error: start: 0x0x416d6801000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d6bff000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d7002000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d700f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d701f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d702f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d703f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d704f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d705f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d706f000, csize: 0x1000, err: -1
mimalloc: warning: mprotect error: start: 0x0x416d707f000, csize: 0x1000, err: -1
error: Could not compile `rustc`.

Caused by:
  process didn't exit successfully: `/home/daan/dev/rust-mimalloc/build/bootstrap/debug/rustc --edition=2018 --crate-name rustc src/librustc/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -C opt-level=2 -C metadata=5e0d3eeff150bb82 -C extra-filename=-5e0d3eeff150bb82 --out-dir /home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps --extern arena=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libarena-af9d93ef6dc08aee.rmeta --extern backtrace=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbacktrace-12b311ad047821a8.rmeta --extern bitflags=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbitflags-0185f8c31122856c.rmeta --extern byteorder=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libbyteorder-84431642a2d21986.rmeta --extern chalk_engine=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libchalk_engine-62466d355dbdc99a.rmeta --extern fmt_macros=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libfmt_macros-482d5e66bb772b92.rmeta --extern graphviz=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libgraphviz-f6a7dc011264b85b.rmeta --extern jobserver=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libjobserver-c8b46a25bb9da522.rmeta --extern log=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/liblog-bd0a443870ca05a1.rmeta --extern measureme=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libmeasureme-091ecbacf269a237.rmeta --extern num_cpus=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libnum_cpus-88fd2b8b876d6e21.rmeta --extern parking_lot=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libparking_lot-f9292c52c4e814ca.rmeta --extern polonius_engine=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libpolonius_engine-e88075d624331b85.rmeta --extern rustc_rayon=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_rayon-ea44b91f88f116f1.rmeta --extern rustc_rayon_core=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_rayon_core-7809f22d4704262f.rmeta --extern rustc_apfloat=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_apfloat-b58383702b0e9127.rmeta --extern rustc_data_structures=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_data_structures-8fd90ba391486a46.rmeta --extern errors=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_errors-8413b71d09c551f0.rmeta --extern rustc_fs_util=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_fs_util-2b07fd141255082a.rmeta --extern rustc_macros=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps/librustc_macros-35a0cb747106834a.so --extern rustc_target=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/librustc_target-a02958114c225952.rmeta --extern scoped_tls=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libscoped_tls-484e840675fbe6fc.rmeta --extern rustc_serialize=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libserialize-f09f146ed063c0de.rmeta --extern smallvec=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsmallvec-739f6fdb85f68aa2.rmeta --extern syntax=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax-0fd6e4d7aa6beaaa.rmeta --extern syntax_pos=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/libsyntax_pos-255f37cb2a800a63.rmeta -Zbinary-dep-depinfo -L native=/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/build/backtrace-sys-1d74c3d84afd8f29/out` (signal: 11, SIGSEGV: invalid memory reference)
command did not execute successfully: "/home/daan/dev/rust-mimalloc/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "8" "--release" "--features" "jemalloc" "--manifest-path" "/home/daan/dev/rust-mimalloc/src/rustc/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit code: 101
failed to run: /home/daan/dev/rust-mimalloc/build/bootstrap/debug/bootstrap build
Build completed unsuccessfully in 0:27:14
Makefile:18: recipe for target 'all' failed
make: *** [all] Error 1

[daanx]

Now, I am trying to run the last failed command in the debugger (/home/daan/dev/rust-mimalloc/build/bootstrap/debug/rustc ...) but I cannot as it expects all kinds of environment definitions, like RUSTC_STAGE, RUSTC_REAL, RUSTC_LIBDIR etc. Does anyone know how to set these correctly? The goal is to run the last command under a debugger and set breakpoints at the warnings in mimalloc and see a stack trace at the fault.

[gnzlbg]

@daanx I don't know how to do that, but we should document how to do that: there is a tracking issue for documenting this here (https://github.com/rust-lang/rustc-guide/issues/443), and a solution for a different case which sadly I don't think will work here.