Open bkolobara opened 2 years ago
Thanks for the report on this! I completely agree the settings are confusing, even as the one who tried to document them all originally I still struggle and reread the docs every time... I think what you've proposed actually makes a lot more sense and would be great to implement. I think we can probably keep wasmtime's internals the same they are now but it should be much easier to consume with the API you've proposed (or something similar). Would you be up for making a PR for this?
For the first example of yours, though, I think Wasmtime is performing as expected. AFAIK there's no OS-specific behavior of Wasmtime itself but I think you're running into OS-specific limitations and behavior of mmap
itself. Using strace
to follow the programs:
So for the first two programs I think that Wasmtime is actually behaving as expected, although perhaps the OS is not behaving as you expect? I am not personally very familiar with the nuances of mmap
on OS-es and how it relates to overcomit and such. For Linux I see that the failing syscall is mprotect
where we make the initial pages read/write. Otherwise though fixing the issue where a static memory is "forced" but we still pick dynamic I think would be fixed with an easier-to-understand configuration scheme like you're proposing. The specific bug here is that instantiation of a 10-page memory into a 5-page-maximum config should simply fail, not accidentally fall back to dynamic allocation.
Thank you Alex for investigating this!
You are right, this is a linux specific setting and Wasmtime is behaving as expected. The setting is vm.max_map_count
and can be increased (until the next reboot) with sysctl -w vm.max_map_count=262144
.
I can try to contribute both PRs, one fixing the actual bug:
The specific bug here is that instantiation of a 10-page memory into a 5-page-maximum config should simply fail, not accidentally fall back to dynamic allocation.
And one for the memory config simplification.
Awesome, sounds great, and thanks!
I'm running into some troubles with the
Engine
configuration settings around memories inwasmtime = "0.33"
. There may be a few different issues here, but because everything is entangled together I will post them all here.I'm trying to spawn 100k wasm instances and am hitting a few problems that I assume are related around virtual memory exhaustion.
This is a minimal example that demonstrates the first issue:
My reasoning here is:
config.static_memory_forced(true)
will force the engine to always use static memory.config.static_memory_maximum_size(15 * 65536)
will allocate up to 15 wasm pages of virtual memory.config.static_memory_guard_size(65536)
will add one more wasm page of virtual memory.This works fine on
64bit MacOs
, but fails to finish on64bit linux
with an error:I have also unsuccessfully tried to use dynamic memories instead:
I assume this should force all memories to be dynamic (
static_memory_maximum_size(0)
), but only allocate up to 15 wasm pages of virtual memory initially. However, this fails again with anInsufficient resources
error onLinux
, but works onMacOs
.What would be a correct approach here that lets me specify a maximum virtual memory size, but also works on
Linux
so I can spawn more than 16k memories?While experimenting with this, I also noticed another inconsistency. If we change the
config.static_memory_maximum_size
to a lower value than the wasm module is requesting:The
Engine
is going to use adynamic
memory, even it was configured withstatic_memory_forced(true)
. This previous example will be failing also onMacOs
too with anInsufficient resources
error. Only if we add aconfig.dynamic_memory_reserved_for_growth(65536);
it will work.Personal thoughts
There is a lot of documentation around these settings, but I have read multiple times through it and somehow still can't figure it out. The most annoying issue is that the documentation only lists defaults for
32 bit
and64 bit
machines, but the behaviour also seems to differ depending on the OS you are running.With
memory64
anddynamic_memory_reserved_for_growth
the lines are even more blured betweenstatic
anddynamic
memory. I'm wondering if maybe we could get away with just one memory model that makes it clearer what is happening in the background?It would be a
dynamic
memory with the following settings:memory_initial_virtual_size(u64)
- The dynamic memory can grow until this point without copying.memory_virtual_resize(bool)
- This would turn it into a static memory.memory_guard_size(u64)
- The guard page.The defaults (64bit systems) would be:
Resulting in the same properties the current defaults have.
This model is less feature rich, but I feel like the current system is so complicated that it's almost impossible to reason about what is going to be happening when you mix multiple settings.