Use-case for WebGPU with WASI

[nasso]

As discussed in #53, WASI needs some way of doing computation and rendering on the GPU, and WebGPU seems to be the best candidate for a cross-platform and safe API. Many potential use cases have been discussed, but I've been suggested to open an issue to describe mine, which is a little bit special.

Basically, I'm building a plugin system based on WASM (similar to this one), where a plugin is just a WASM binary that implements some graphics (and/or compute) logic. The goal is to be able to load these plugins in the browser (with the polyfill) or in a native WASM VM (like Wasmtime) without any changes. WASI allows giving plugins a common interface without relying on a specific environment and controlling their capabilities securely.

Here are the basic principles of my use-case:

• The host creates the WebGPU instance and device (e.g. WebGPU on the web, wgpu or dawn on native).
• Plugins are sand-boxed and are not necessarily trusted code (can be third-party).
• A plugin has one or several inputs and outputs (textures).
• Plugins must be able to share resources (only if given access to it directly, e.g. as an input). That implies sharing the WebGPU device (and other objects) created by the host.
• It must be possible to "chain" plugins (take the output of plugin A, pipe it to the input of plugin B and so-on) with little to no overhead (this will happen a lot, avoiding copies would be good).
• A plugin doesn't need to do any presenting (it only works from and to off-screen textures).
• Only the web host has to actually present anything to the screen.
• The native host will usually just save the rendered frames to disk.

I should probably add that all this is offline rendering (as opposed to real time rendering), so speed is important, but not critical.

If I forgot anything valuable or if anything is unclear please let me know, I'll be happy to provide more details!

[sunfishcode]

Since you mention wanting to run the same wasm binary in both the browse and non-Web VMs, this will likely want to be based on the WebIDL specification of WebGPU (as opposed to the C API), as that's what's exposed in browsers. The Interface Types proposal is expected to provide the foundation needed to build tools to integrate with WebIDL.

[olanod]

Interesting use-case, actually when I created #53 I was planning to achieve pretty much what you describe and perhaps a bit more. I was working on self-driving cars stuff that is now paused since I changed to a different industry but the basic idea is to implement Wapr(nothing to see there ATM), I'll leave the idea out there just in case is useful.

A Gstreamer inspired library/framework where a directed graph defines how data flows from node to node, each node is a WASI binary that is auto loaded by the framework using a URL or resolving the URL from a registry/market place, the underlying system could be an actor framework that guarantees delivery of messages to the different components, queues them, adds back-pressure, restarts the node if it fails, etc. and supports remote actors so parts of the graph can run in different devices, the idea being, early stages of self-driving technology requires getting lots of data coming from different sensors(#251) and most or all computing being done in the car and a screen(#174) or user device only getting the final output to be rendered but in the future those plugins can gradually and transparently be moved to run on a the user device like in a web app running on a smartphone for example hoping some day user is in complete control of their dumb car and not at the mercy of an evil automaker ;P

On the plugin side initially data can be received and sent just using the current support for stdin/stdout interfaces, perhaps with specialized WASI APIs in the future that handle automatically the multiplexing of streams of data coming directly from a socket or something(WASI libp2p?), once the data comes you'll want to process before passing it through and that's where having threads, GPU or even neural network(#59) APIs could come handy, if the plugin is written in Rust for example, I imagine that using wgpu-rs would already take care of using the WebIDL or expect the binary to get compiled using the standardized WegGPU-like API that the host runtime(e.g. browser) exposes and could use the WebIDL under the hood. It would also make sense that the framework itself compiles to WASI so to run WASI within the host should expose its WASI engine(#149) and in the browser would be nice to just create a new WasiWorker('pipeline.wasm')(#297) and the whole system runs separately in the background spawning WASI actors/tasks on demand in different threads. ... That more or less concludes my hopes for WASI, I hope I'll get the time to experiment a bit more with the ideas or contribute to projects making them a reality 😅

[ShadowJonathan]

(WASI libp2p?)

I would personally heavily recommend against libp2p, as they have (in my opinion) foregone their original mission and idea of implementing a generalized p2p network interface, and instead are using it primarily for IPFS' use-cases, adding cruft by "setting" specific behaviours where it otherwise could've been generalised, for which it is then not possible anymore to remove them (due to the fact nearly all other nodes on the network would behave that way.). Even if they'd change their stance on it tomorrow, the current network is thus not very adaptable to those changes. Their management have so shown to have no interest to invest in such low-level longevity, only focusing on the highest-level aspect; IPFS and Filecoin.

Sorry if I come over a bit harsh, biased, or blunt, but those are my feelings regarding WASI considering libp2p as a generic low-level interface. At the very least, I advise caution when considering putting it as a fundamental building stone.

WebTransport looks much more interesting in comparison, imo.

[dvc94ch]

Anyone that thinks libp2p, ipfs or filecoin don't suck has only read PL propaganda and never actually used it or had to use it... I actually worked with these for years and I'm happy to clarify if anyone wants a detailed review of any of those...