Creating heaps in pre-allocated arenas

Hi! Our team has been trying to adopt mimalloc for the project we've been working on for embedded devices and consoles. We would like to limit how much memory mimalloc would use on a given system and control that ourselves. There were issues on this subject before (e.g https://github.com/microsoft/mimalloc/issues/617). My takeaway from that thread was that we can create an arena in a specific memory region and use that arena to create thread local heaps using mi_manage_os_memory_ex and mi_heap_new_in_arena. Unfortunately, mi_heap_new_in_arena still tries to create an arena to create the heap (here), which doesn't work for us.

As it was discussed in other threads, we would like to limit all mimalloc ops to a single arena that we create and provide to mimalloc and if that's exhausted, just return NULL and we would handle OOM. Is there a way for mimalloc to use the arena that we create to allocate the heap within that arena, instead of allocating a different one?

Looking deeper into the mi_heap_new_in_arena I was able to figure out where that extra arena allocation was coming from, it was allocating the memory for the initial backing heap which is used to create a new heap, please correct me if my understanding is wrong here.

I was somewhat able to bypass this issue by setting bheap->arena_id in mi_heap_new_in_arena to the arena id I'm passing, i.e:

mi_decl_nodiscard mi_heap_t* mi_heap_new_in_arena(mi_arena_id_t arena_id) {
  mi_heap_t* bheap = mi_heap_get_backing();
  bheap->arena_id = arena_id; // <<<<<<<<<<<------------------ This line
  mi_heap_t* heap = mi_heap_malloc_tp(bheap, mi_heap_t);  // todo: OS allocate in secure mode?
  if (heap == NULL) return NULL;
  // don't reclaim abandoned pages or otherwise destroy is unsafe  
  _mi_heap_init(heap, bheap->tld, arena_id, true /* no reclaim */, 0 /* default tag */);
  return heap;
}

My understanding is that arenas are shared between threads and this seem to be an acceptable change modification to scope everything to a single arena?

With the change above, the only OS call would happen when a new thread tries to call mi_heap_new_in_arena and _mi_thread_heap_init ends up calling the OS to allocate memory for the new TLS backing heap. Is backing heap only used to hold data for the heap being created? I wonder if this part could be somehow optimized as well.

There's one moment with _mi_thread_heap_init when it allocates mi_thread_data_t from the OS, it rounds its size to 16KB, asking for x4 memory.

I've tried the changes above, with disabled MI_STATS, and was able to minimize the number of OS calls from mimalloc and so far things seem to run smoothly. Does this modification breaks mimalloc architecture in some way that I'm missing?

Thank you!

microsoft / mimalloc

Creating heaps in pre-allocated arenas #937