Some attempts to combine WASM and gVisor

[lubinszARM]

When I tried to apply gVisor to more scenarios in production area, I found that gVisor has the potential to be the guest user space kernel of WASM runtime. In addition, when I attended the CNCF runtime conference, I found that WASM became more and more popular as the CNCF runtime.

So, I made a simple POC to verify my idea: https://github.com/lubinszARM/gvisor/tree/pr_platform_wasm_poc

In my simple poc, I added a WASM platform and a simple wasm syscall interface support(fd_write). Full wasm syscall interface list is here: https://docs.rs/wasi/0.10.2+wasi-snapshot-preview1/wasi/wasi_snapshot_preview1/index.html We can follow the steps below to verify my poc: 1, add a new runtime in /etc/docker/daemon.json "myrunsc": { "path": "/usr/local/bin/runsc", "runtimeArgs": [ "--platform=wasm" ] },

2, docker run --rm --runtime=myrunsc younglook/wasi-hello

3, verify the result

# docker run --rm --runtime=runsc younglook/wasi-hello
hello world

4, hello.wat is like following:

(module
    ;; Import the required fd_write WASI function which will write the given io vectors to stdout
    ;; The function signature for fd_write is:
    ;; (File Descriptor, *iovs, iovs_len, nwritten) -> Returns number of bytes written
    (import "wasi_snapshot_preview1" "fd_write" (func $fd_write (param i32 i32 i32 i32) (result i32)))

    (memory 1)
    (export "memory" (memory 0))

    ;; Write 'hello world\n' to memory at an offset of 8 bytes
    ;; Note the trailing newline which is required for the text to appear
    (data (i32.const 8) "hello world\n")

    (func $main (export "_start")
        ;; Creating a new io vector within linear memory
        (i32.store (i32.const 0) (i32.const 8))  ;; iov.iov_base - This is a pointer to the start of the 'hello world\n' string
        (i32.store (i32.const 4) (i32.const 12))  ;; iov.iov_len - The length of the 'hello world\n' string

        (call $fd_write
            (i32.const 1) ;; file_descriptor - 1 for stdout
            (i32.const 0) ;; *iovs - The pointer to the iov array, which is stored at memory location 0
            (i32.const 1) ;; iovs_len - We're printing 1 string stored in an iov - so one.
            (i32.const 20) ;; nwritten - A place in memory to store the number of bytes written
        )
        drop ;; Discard the number of bytes written from the top of the stack
    )
)

After communicating with my colleagues from the wasm team, we summarized the benefits of this approach: 1, Can meet the requirements of OCI https://github.com/bytecodealliance/wasmtime/issues/358 2, Convert system call into function call to get better performance, the specific WASI link method is as follows: https://github.com/lubinszARM/gvisor/blob/pr_platform_wasm_poc/pkg/sentry/wasmvm/wasmtimevm.go#L39

There are some similar projects that use this method, such as: https://github.com/kenny-ngo/wasmjit https://github.com/wasmerio/kernel-wasm

3, Introduce a high-performance network, such as DPDK 4, Introduce a user mode kernel for wasm runtime to improve security

[lubinszARM]

/cc @avagin

[amscanne]

I love this. I'm a big fan of wasm and have wanted to this for a long time, it's awesome that you've put together a POC.

Here are some thoughts:

• Rather than a separate platform, I actually thought we could just support native WASI binaries that are in container images. E.g. interpret the WASI header and hook in to the loader itself. Some minor refactoring would be required to allow a loaded binary to use a different platform however, but I think it's fine. (For example, the LoadArgs could come with the current platform, and the top-level "Load" function could return a platform that should actually be used for that task.
• We need a better way to hook into the wasmtime Memory subsystem. We would be able to provide a linear byte slice for a given memory size (potentially stitching together multiple pgalloc chunks in the address space), but they need to be sourced from the pgalloc file for things like memory accounting to work correctly. This would mean that the "grow" operation would effectively need to be able to ask us to change it. I'm not sure what the wasmtime function is doing there, but it's probably a trivial reallocation.
• We would need a way to occasionally drop into the regular task.Run loop for WASI tasks. This might be something that works with regular pre-emption, but it I believe that would depend on the implementation in wasmtime. Not sure what the right answer for this is.

The only big concern / problem that I see needing to be resolved is the CGo issue. The current wasmtime bindings are great (they even use bazel already, so can just be plugged into the workspace) but they will make runsc dynamically linked and change the system surface. I wonder if it's possible to link that pile of code statically? E.g. what are the symbols required by wasmtime? Maybe we could even provide that. E.g. if we can provide them with a "malloc" symbol, we could do allocation from the pgalloc file! Then we wouldn't even need to hook into the wasmtime API bindings, and it would account for all memory consumed by the wasmtime runtime! Anyway, curious to hear your thoughts.

[lubinszARM]

https://v8.dev/blog/emscripten-standalone-wasm

[lubinszARM]

Hi @amscanne I agree with your suggestions. If wasmtime-go is to be integrated, there are still many problems that need to be resolved.

Currently I am still discussing with my colleagues the benefits of putting this combination in serverless/faas scenario, and trying to find a suitable wasi program.

I agree that memory management is indeed a problem for this POC. I searched some information, maybe I can try 'emscripten_notify_memory_growth'. :)

[codefromthecrypt]

Hi, there. I am curious if it is a strict requirement to use wasmtime-go? If not, you might consider wazero, which is the only WebAssembly runtime written in Go (even has JIT support). It would be cool to continue this work without the CGO roadblock, and maybe this helps https://wazero.io/

Disclaimer, I'm an active dev on wazero, though that may be helpful if you are interested in proceeding as you can nag me. We will be cutting a beta API soon, so now's a good time to practice, and we'd appreciate any experience you can give.

[tanjianfeng]

@codefromthecrypt Unfortunately, @lubinszARM is not on this position anymore, not sure if he's still interested in this topic.

Personally, I'm still not sure why people use wasm out of browsers. I may fail to catch the essence of this potential, "a common assembly language for the client, servers, IoT devices and more".

[codefromthecrypt]

@tanjianfeng thanks for the note. For what you are mentioning, I think main gain for go folks is having a statically compiled binary and also being able to load extensions (without CGO). I can't speak to effectiveness of FaaS etc like krustlet, but there are some arguably effective integrations (like in envoy wasm instead of lua or in some DBs wasm as a UDF). this is semi-on topic as it speaks to the motivation of doing anything here.

Certainly people are a bit amped about it recently, though yeah is it hype or interest? I'll leave it with you to decide. https://news.ycombinator.com/item?id=31405890

In any case, if you want a hand, ping me. Otherwise, I won't spam further! Thanks again for the reply.

[tanjianfeng]

Many thanks for the information. If I understand it correctly, it's similar to what eBPF means to linux kernel. The body softwares are willing to add extensions/UDFs in a flexible way but protect any possible bad effects along with.

Some of my colleagues want to add extensions into mosn; I will recommend wazero to them.

Further, as we introduced a C language module in gVisor sentry, two captious questions just fly into my head:

• Does this reduce the overhead of FFI? CGO takes about 60~70ns per call.
• Some extensions/libs are written in arch-native way (SIMD, cache-align, cache ping pong avoidance, ...), after turning it to wasm, and back to machine code, perf will drop?

[codefromthecrypt]

Some of my colleagues want to add extensions into mosn; I will recommend wazero to them.

Thanks. I think others had been interested in this also. OTOH, now is a better time to action things as we'll have a beta api cut soon.

Does this reduce the overhead of FFI? CGO takes about 60~70ns per call.

There are so many things about CGO it almost needs a book. Overhead is one of them. We have benchmarks that rank operations in our relatively new runtime vs established ones, and things that execute callbacks usually win (if using our compiler runtime) even if our API for memory access has guards not all runtimes have. Anyway, we put benchmarks so people can see for themselves, and also there is an excellent project by @wuhuizuo which looks at different patterns, too. https://github.com/wuhuizuo/go-wasm-go

Some extensions/libs are written in arch-native way (SIMD, cache-align, cache ping pong avoidance, ...), after turning it to wasm, and back to machine code, perf will drop?

WebAssembly is a conceptual virtual machine, which has a hardware bias in its ISA. So, unlike JVM, there are instructions for SIMD. However, like JVM the ISA is decoupled from the actual platform. This means there are three issues.. which low level features have been developed, which are supported by a compiler to wasm, and which are supported by the runtime. Both sides of translation are undocumented by spec so room for interpretation and optimization. For example, it happens to be the case that some instructions in Wasm closely resemble arm64, except modeled as stack machine not register machine.

Anyway, so for very high performance code, you might choose a different source language than for example TinyGo which doesn't emit SIMD instructions. You can look at https://emscripten.org for some tips.

Then, on the runtime even if wazero will soon complete WebAssembly 2.0, there are unfinished feature proposals you might want, that the compiler supports, but wazero doesn't yet. Or maybe you notice how we lower our IR to native instructions isn't ideal.

The good news is that any changes to the runtime are easy to accomplish and test because wazero is pure go and has no dependencies. You can fork and test and help via pull request using normal Go flow. no shared library deployment complexity for example. It is just go.

Hope this helps!

[amscanne]

Sorry for the delay, I am super interested in this and actually checked out wazero a few months ago. It's awesome!

One question: for integration with gVisor (what I was actually considering a few months ago), we would need the ability to hook at various platform points. For example, memory allocation (and resizing) operations. I believe this is mostly an API change (and plumbing relevant bits down). Do you think this would be a friendly change or are there immediate concerns?

[codefromthecrypt]

@amscanne thanks for the kind words about wazero.

About memory plumbing: Right now, we have apis for config and runtime, but not allocation of the underlying memory itself. All apis in wazero are currently implemented only by wazero itself.

• api.MemorySizer - allows overriding max memory or allocated capacity
• api.Memory - runtime api for accessing memory including requesting to grow it.

So, I think you are asking for a plugin to actually control the underlying memory, and to be able to externally provide the impl of that. My guess is this would be a factory function that returns an api.Memory or an alternative way to allocate the slice underneath of it

Ex. here's the current internal function to generate the initial memory instance

func NewMemoryInstance(memSec *Memory) *MemoryInstance {
    min := MemoryPagesToBytesNum(memSec.Min)
    capacity := MemoryPagesToBytesNum(memSec.Cap)
    return &MemoryInstance{
        Buffer: make([]byte, min, capacity),
--snip--

I think the main edge cases will be about design choices that invalidate memory access patterns in assembly code generated by our compiler. As long as it can think as if it were a go []byte, there's maybe no changes. If changes are needed, I think it is worth pursuing even if it ends up a special hook only used for gVisor. It seems like if the main concern is only memory allocation (initial and on grow) it won't be fruitless to investigate.

@mathetake any other thoughts?

[mathetake]

no immediate concern from me! Let's see the concrete necessary API changes and how they can be implemented in our compiler etc. Exciting!

[lubinszARM]

Sorry for delay. @tanjianfeng @amscanne Long time no see. After a busy period of cross-country moving, I'm glad to see someone interested in this topic. Exciting! ：）

[mathetake]

Hi there, the wazero runtime author here. I spent yesterday and actually ported the PoC of @lubinszARM to work with wazero and played around with some gVisor's code base. So here's my take; I cannot imagine the Wasm/WASI support would go beyond a toy. The reason is that at the end of the day the Wasm platform implementation won't be able to leverage the gVisor-specific concepts, but rather it would just become the thin implementation of OCI runtime inside the gVisor APIs tightly coupled with Linux concepts. The other reason is that WASI is completely half-baked (though there will be v2 WASI, but I believe it would take a couple of years more to reach stability), plus lack of threading/atomics/signaling semantics in the Wasm specification. So tldr is that I don't think it would be worth the effort, given the status-quo of Wasm/WASI specs and the gVisor's internal coupling with Linux (or "real" OSes). But perhaps in the future, we should revisit this once Wasm/WASI becomes more mature vs now. Thanks!