Crash in _mi_heap_delayed_free in Windows 10 (64 bit)

[ArsMasiuk]

Since v.2.0.3 the application which uses mimalloc crashes sporadically mainly at _mi_heap_delayed_free() function. The application has been compiled with VS2017 and Qt 5.15.2 (64 bit). Updating to 2.0.4 did not solve the issue unfortunately. Partial stack trace attached.

[daanx]

Ouch -- that is not good.

• You say since v2.0.3 -- what version did you use before? (was it v2.0.2 or a v1 version ?)
• What is the error? NULL pointer dereference, dereference in uncommitted memory, etc?
• Can you reproduce it? or is it only sporadically? How often?
• If you can somewhat regurlarly reproduce it, is it possible to test with MIMALLOC_ALLOW_DECOMMIT=0 in the environment and see if it still occurs. (if it does not, I have at least a good idea what might be wrong)

Best, Daan

[ArsMasiuk]

I think it was v2.0.1 - but I cant be 100% sure.. The error is sometimes "Heap Corruption" and sometimes "Access violation reading location 0xFFFFFFFFFFFFFFFF." It appears quite sporadically when the RAM consumption of the application goes towards 1GB. And what is MIMALLOC_ALLOW_DECOMMIT=0? is it compiler definition so it should be among "DEFINES" ?

[daanx]

Hi @ArsMasiuk, hmm, maybe it is an earlier heap corruption or buffer overflow. I think we should first try to run with full debug info:

1. compile and link mimalloc in debug mode: (e.g. cmake ../.. -DCMAKE_BUILD_VARIANT=Debug -DMI_DEBUG_FULL=ON ), and run your program with that. This will be slower but will catch most buffer overflows, double frees, etc.

2. if that shows no issues, then run with the environment variable MIMALLOC_ALLOW_DECOMMIT=0 -- this will prevent mimalloc from decommitting memory; This is just to see if the issue would still occur in that case (to give me a clue what could be going wrong).

Thanks.

[ArsMasiuk]

It looks like mimalloc 2.0.4 (from trunk) causes a memory leak somewhere. With 2.0.4, RAM consumption was over 2 GB per process within our application and tended to grow. With mimalloc 2.0.3, RAM usage was always constant and never went over 1 GB.

[daanx]

Hi @ArsMasiuk;

It looks like mimalloc 2.0.4 (from trunk) causes a memory leak somewhere.

Ah, I was changing a lot yesterday doing experiments. What commit did you use? (use git log to see it).

(ps. and if you use MIMALLOC_ALLOW_DECOMMIT=0 it is expected to be more of course -- that setting is just to see what causes the original issue)

[Jarred-Sumner]

I'm seeing a similar crash when using several arenas in different threads that free data allocated from a mi_heap_t. It seems very much like a race condition.

This is on the Mimalloc dev-slice branch. It reproduces in both debug builds and in release builds (the screenshot is a debug build). I put mi_heap_collect in a number of places to attempt to find where it happens earlier. It does not happen consistently. I am not confident that my own code isn't at fault here. But I did add some logging to make sure that mi_heap_t only allocates in the expected thread.

_mi_heap_delayed_free_partial(mi_heap_s*) (/Users/jarred/Code/bun/src/deps/mimalloc/src/page.c:317)
_mi_heap_delayed_free_all(mi_heap_s*) (/Users/jarred/Code/bun/src/deps/mimalloc/src/page.c:303)
mi_heap_collect_ex(mi_heap_s*, mi_collect_e) (/Users/jarred/Code/bun/src/deps/mimalloc/src/heap.c:148)
src.mimalloc_arena.Arena.gc (/Users/jarred/Code/bun/src/mimalloc_arena.zig:196)
src.bundler.bundle_v2.LinkerContext.generateChunkJS_ (/Users/jarred/Code/bun/src/bundler/bundle_v2.zig:3914)
src.bundler.bundle_v2.LinkerContext.generateChunkJS (/Users/jarred/Code/bun/src/bundler/bundle_v2.zig:3731)
src.thread_pool.Do__anon_55362.Runner.call (/Users/jarred/Code/bun/src/thread_pool.zig:299)
src.thread_pool.ConcurrentFunction((function 'call')).Routine.callback.27 (/Users/jarred/Code/bun/src/thread_pool.zig:221)
src.thread_pool.Thread.run (/Users/jarred/Code/bun/src/thread_pool.zig:642)
Thread.callFn__anon_269626 (/Users/jarred/zig/0.11.0-dev.1934+10c74631b/files/lib/std/Thread.zig:404)
Thread.PosixThreadImpl.spawn__anon_199293.Instance.entryFn (/Users/jarred/zig/0.11.0-dev.1934+10c74631b/files/lib/std/Thread.zig:677)
_pthread_start (@_pthread_start:40)

[daanx]

Thanks for the report! Yikes -- I hope we can figure out what this is. The _mi_block_nextx just reads the next field in the linked list of delayed free nodes -- a crash may be caused by a 'use-after-free' or 'buffer overflow' as well (although, debug or secure mode should pick this up without crashing). Wish I can find some way to repro -- now extensively testing with TSAN but no luck so far.

When you say "arenas", do you mean you reserved mimalloc arena's ? Or you mean just multiple threads freeing in each others heaps?