Insufficient memory error when spawning too many processes

[drogus]

I get a Insufficient resources: System call failed: Cannot allocate memory (os error 12)' error when trying to spawn too many processes. The processes are almost empty - just a single sleep instruction is in each of them. I've tried setting the max memory to a high number using the Config struct, but it only got me ~16k processes. Let me know if you need any more info, but the code is basically this:

use lunatic::process::sleep;
use lunatic::{process, Config, Environment, Mailbox};

#[lunatic::main]
fn main(m: Mailbox<()>) {
    let mut config = Config::new(10_000_000_000, Some(u64::MAX));
    config.allow_namespace("");
    let mut env = Environment::new(config).unwrap();
    let module = env.add_this_module().unwrap();

    module
        .spawn_link(m, |_parent: Mailbox<()>| {
            for n in 0..1000000 {
                process::spawn_with(n, handle).unwrap();
            }
        })
        .unwrap();
    sleep(1000000);
}

fn handle(n: u64, _: Mailbox<()>) {
    println!("Spawn: {}", n);
    sleep(1000000000)
}

What's interesting is that if I try set the memory limit higher than that (like 100B or I also tried u64::MAX), it either crashes right away or very early. With the 100B example it crashed while trying to allocate 100GB of memory. Even though it's the case tuning the memory down didn't help either - I tried values just high enough to create anything at all and it didn't really get me anywhere either.

[bkolobara]

I think that this is more of a documentation issue. The max memory configuration of the Environment limits the maximum memory of each process spawned into the environment. It doesn't set the total environment limit.

We preallocate a lot of virtual memory for each process, up to the maximum size. Virtual memory is mostly free on 64bit architectures, but it's not completely free. If you allocate for each process 10 Gb of virtual memory you are going to reach 128 Tb usage around 12k processes. This is usually the OS limit.

If you set a lower per process limit you can easily spawn 1M processes:

use lunatic::process::sleep;
use lunatic::{process, Config, Environment, Mailbox};

#[lunatic::main]
fn main(m: Mailbox<()>) {
    let mut config = Config::new(10_000_000, None);
    config.allow_namespace("");
    let mut env = Environment::new(config).unwrap();
    let module = env.add_this_module().unwrap();

    module
        .spawn_link(m, |_parent: Mailbox<()>| {
            for n in 0..1000000 {
                process::spawn_with(n, handle).unwrap();
            }
        })
        .unwrap();
    sleep(1000000);
}

fn handle(n: u64, _: Mailbox<()>) {
    if n % 1000 == 0 {
        println!("Spawn: {}", n);
    }
    sleep(1000000000)
}

[zhamlin]

With the above code I'm getting the same error around 16K processes as well. Insufficient resources: System call failed: Cannot allocate memory (os error 12)',

Not sure if it matters but I'm running this on Linux.

[bkolobara]

Thanks @zhamlin for testing this. You are right, there is an issue, but only on linux. I have opened an issue with Wasmtime to figure out what the correct settings here are: https://github.com/bytecodealliance/wasmtime/issues/3695

[zhamlin]

Increasing vm.max_map_count allowed me to spawn more processes.

sysctl -w vm.max_map_count=262144

[bkolobara]

Yes, this setting did the trick. 🎉 Thanks again @zhamlin for looking into it.

Linux has a limit on how many memory maps it can hold per OS process and each lunatic process (wasmtime instance) crates a few (mmap/mprotect calls for virtual stacks + linear wasm memory). I think it's 4 in total, and taking into account that vm.max_map_count is by default usually set to 64k it results in a 16k process limit on linux.