Add forking capabilities to clone()

[rennergade]

In addition to creating threads via clone() we also want to fork processes. Wasmtime doesn't have this capability built in but wasmer seems to have it: https://github.com/wasmerio/wasmer/blob/eb9127036add9f2a174ec1623600ebcc802fed6f/lib/wasix/src/syscalls/wasix/proc_fork.rs#L23

Can we look into what they're doing here and see if we can recreate this as an add-on module to wasmtime.

[syrusakbary]

Why not using Wasmer? :)

[qianxichen233]

Some updates/docs for fork: We have successfully implemented fork inside wasmtime, using the same approach from Wasmer. Detailed steps for current fork implementation:

1. When the parent calls fork and transfers the control to wasmtime, it calls asyncify_start_unwind from the parent module
2. The parent module will unwind until it returns from _start function. We catched the return here, and make a callback to call asyncify_stop_unwind to stop the unwind process, and retrieves the unwind_data.
3. re-initialized a new instance with PreInstance for child. Copied the entire memory from parent to child (unwind_data is also copied to child here).
4. For both parent and the child, call asyncify_start_rewind to mark the start of rewinding. Then calls the _start function of the module to start rewind. This will make the parent and child finally reach to fork again. This is catched here, by checking the unwinding state in Store that was set before asyncify_start_rewind. This new if separate branch will call asyncify_stop_rewind and return. After fork returns, both the parent and the child should be restored to the breakpoint where fork is called and can continue its normal execution.

Besides, we also have a very basic support for forking inside multi-threading environment. The way memory of threading works in wasmtime is that the wasm module must declare the memory as imported and shared when compiled, so that wasmtime could create a new shared memory and put it into the Linker struct. In this way, whenever the wasm module wants to access the memory, it will look for the imported memory from the Linker. And since the linker is shared between threads, the memory is also inherently shared. In order to make this system working with fork, we cloned the Linker for the new forked instance, and replaced the imported memory with a new memory. However, there is still issues in fork interacting with threads.

1. Forking when the process has multiple threads running at the same time may cause potential issue that the state of the process is not correctly copied to the child. This is because when one of the thread in the process calling fork and duplicate its state, other threads are still running and may modify the process state, causing the state copied to the child may be slightly different from the state of the parent.
2. Forking inside non-main thread is not supported right now. However, this is not hard to implement (if ignore the previous issue), since we can just call wasi_thread_start function instead of _start function during the rewind process and it should works fine.

[yzhang71]

https://docs.google.com/document/d/1KIuQdGLL3cAQ_lv08rQvDt0-TCeI8dn8sChrE3sQR8U/edit#heading=h.ki2aav796ny6

[rennergade]

Great job at getting most of this together. One thing I want to hark back on is what the title of this issue suggests, which is fork should be called by clone() syscall out of libc, which also should be the call for creating threads. We should use basically the same semantics as clone() in Linux.

[rennergade]

Blueprint:

1. User program calls fork() or pthread_create -> libc
2. In libc, pthread_create should call clone_internal, or fork() calls arch_fork() which calls clone_internal. Youll need to package the flags and arguments correctly.
3. Call from libc clone_internal via MAKE_SYSCALL to RawPOSIX dispatcher
4. Add clone_syscall in RawPOSIX, change current fork_syscall to fork_helper. Only call fork_helper if flags indicate it is a fork.
5. Call from RawPOSIX clone_syscall back to wasmtime runtime to do system setup (ie wasi_thread_spawn) return back to RawPOSIX.
6. Return back to libc from RawPOSIX as normal.

[qianxichen233]

When I was testing some multi-threading program with large buffer involves, I realized there is an minor issue with our approach of using asyncify to implement thread as a shared-memory version of fork. So basically creating a thread is like doing a fork, but the memory will not be copied and they are using the same memory. And a new stack address is assigned to thread for its use to it won't conflict with the parent's stack. Now the issue is that, the unwind/rewind process will actually save the stack pointer and restore it back after rewinding. This makes the thread's stack pointer forcely restored back to parent's stack pointer even if I already set the child's stack pointer before. Currently I can think of two solutions:

1. the first solution is to modify the copied unwind data for child (unwinding process will generate an unwind data, which is used to do the rewinding) to make the stack pointer stored in unwind data becomes the new stack pointer. This approach might be a little bit hard, since the unwind data is a raw binary, so it is hard to tell where is the stack pointer stored inside it.
2. the second solution is to modify the asyncified wasm module directly, which is what I am currently using. I tried to remove the instruction where stack pointer is restored (replace global.set __stack_pointer with drop), and this seems to be working fine right now. However, although the tests seems to be passed, I am not very sure what exactly could happen using this approach. And this would add another step of compliation into our compliation process.

[JustinCappos]

I'd recommend asking on Zulip to understand more about why it was done in this way in the first place. Your solution seems logical, but I also worry about what is now broken.

On Thu, Oct 3, 2024 at 1:05 PM Qianxi Chen @.***> wrote:

When I was testing some multi-threading program with large buffer involves, I realized there is an minor issue with our approach of using asyncify to implement thread as a shared-memory version of fork. So basically creating a thread is like doing a fork, but the memory will not be copied and they are using the same memory. And a new stack address is assigned to thread for its use to it won't conflict with the parent's stack. Now the issue is that, the unwind/rewind process will actually save the stack pointer and restore it back after rewinding. This makes the thread's stack pointer forcely restored back to parent's stack pointer even if I already set the child's stack pointer before. Currently I can think of two solutions:

1. the first solution is to modify the copied unwind data for child (unwinding process will generate an unwind data, which is used to do the rewinding) to make the stack pointer stored in unwind data becomes the new stack pointer. This approach might be a little bit hard, since the unwind data is a raw binary, so it is hard to tell where is the stack pointer stored inside it.
2. the second solution is to modify the asyncified wasm module directly, which is what I am currently using. I tried to remove the instruction where stack pointer is restored (replace global.set __stack_pointer with drop), and this seems to be working fine right now. However, although the tests seems to be passed, I am not very sure what exactly could happen using this approach. And this would add another step of compliation into our compliation process.

— Reply to this email directly, view it on GitHub https://github.com/Lind-Project/lind-wasm/issues/10#issuecomment-2391908826, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGRODZOJ7NCWORT6BYXQBDZZV2NPAVCNFSM6AAAAABMG7CVHSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJRHEYDQOBSGY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[qianxichen233]

Yeah this approach is not stable and it can crash in some case. So I tried with the first approach and dived deeper into how asyncify modifies the wasm code by reading through its wat file, and I figured out that it looks like the stack pointer is set by a fixed variable in unwind data. I tried to replace the value of the variable in unwind data and now the stack pointer seems to be set correctly and the tests is running stably. However, there is still one remaining question about the offset to that fixed variable. I am not sure if the offset is a fixed value, or it may change for some other program. I've tried several test cases and it looks like 0xc is working for all of them. I am not sure if people on Zulip would like to answer question about Asyncify because that is not something wasmtime is using