microsoft / mimalloc

mimalloc is a compact general purpose allocator with excellent performance.
MIT License
9.74k stars 793 forks source link

Shutdown issues using heap bound to arena, `mi_collect` #828

Open biddisco opened 7 months ago

biddisco commented 7 months ago

We have a custom arena which needs to be thread safe, we allocate memory (with mmap) and use

bool success = mi_manage_os_memory_ex(ptr, size, true, false, true, numa_node, true, &m_arena_id);

In our allocator, we store a thread local heap ptr to access it, and this appears to work fine

    if (!thread_local_ex_mimalloc_heap)
    {
        thread_local_ex_mimalloc_heap = mi_heap_new_in_arena(m_arena_id);
    }
    ... // later when allocating etc
    rtn = mi_heap_malloc_aligned(thread_local_ex_mimalloc_heap, size, alignment);

If we use MI_SKIP_COLLECT_ON_EXIT a simple test program that generates N threads, make a load of allocations and deallocations behaves completely as expected, but if we enable mi_collect to run on exit, then on shutdown we have a segfault when pages are 'cleaned up' (?)

1  std::__atomic_base<mi_segment_s *>::load  atomic_base.h  820  0x7ffff7f69f39 
2  std::atomic<mi_segment_s *>::load         atomic         578  0x7ffff7f69f39 
3  std::atomic_load_explicit<mi_segment_s *> atomic         1324 0x7ffff7f69e45 
4  mi_abandoned_pop                          segment.c      1195 0x7ffff7f6852d 
5  _mi_abandoned_reclaim_all                 segment.c      1384 0x7ffff7f6927a 
6  mi_heap_collect_ex                        heap.c         138  0x7ffff7f57ce6 
7  mi_heap_collect                           heap.c         178  0x7ffff7f57e7e 
8  mi_collect                                heap.c         182  0x7ffff7f57ea6 
9  mi_process_done                           init.c         633  0x7ffff7f5a585 
10 __cxa_finalize                            cxa_finalize.c 83   0x7ffff7845a56 

The error is triggered in segment.c when mi_abandoned_pop tries to get the 'next' segment from the abandoned list(?)

      mi_segment_t* anext = mi_atomic_load_ptr_relaxed(mi_segment_t, &segment->abandoned_next);

Now if we look at the value of th segment pointer, it points to our custom arena memory allocation (in this case segment points to 0x7fffd4000000 which from printf and inspection, I know is the base address of our custom memory arena allocation

        anext   @0x7ffff7e39440 mi_segment_t
        next    0   mi_tagged_segment_t
        segment 0x7fffd4000000  mi_segment_t*
        ts  140736750161921 mi_tagged_segment_t

If we simply do not delete our own custom arena memory on shutdown (munmap), the segfault goes away - because now the seegment is a valid address and all is well. We can also enable MI_SKIP_COLLECT_ON_EXIT to hide the problem - but it seems to me that this is a bug in mimalloc - segments from custom arenas, should not be moved to the primary heap at any time for cleanup operations - they should remain owned and managed only in the heaps bound to those arenas.

Have I misunderstood whats going on, and can we fix our code some other way. Thanks.