[wasm] New failure condition introduced by anonymous mmap implementation

[kg]

The new anonymous mmap implementation ( https://github.com/dotnet/runtime/pull/101871 ) which replaces emscripten's mmap (malloc wearing a fancy hat) introduces a new failure scenario that wouldn't have failed before:

1. Allocate ~2GB of managed objects
2. Release the objects
3. Force SGen to collect garbage enough so that it releases all the now-unnecessary pages it allocated. (It's possible it doesn't actually do this correctly right now...)
4. Try to malloc 1gb of memory

In the old emscripten mmap, all sgen mmap operations and native malloc calls are all serviced by malloc under the hood, so "pages" released by munmap become available to mmap.

In the new mmap, these pages remain controlled by mmap, so emscripten malloc can't use them since it doesn't overlay on top of a real mmap implementation.

At present the workaround for this would be to disable the new mmap implementation for your application (there is a runtime option). In the long run I think the fix is to stop using emscripten malloc, and ship our own dlmalloc or mimalloc that sits on top of our mmap.

[dotnet-policy-service[bot]]

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.