Proposal: WASI Windowing

tareksander commented 4 months ago

Introduction

In case windowing is deemed out-of-scope for #578, there should be a separate proposal for WASI windowing. Given that that proposal is based on bringing WebGPU to WASI and wgpu and WebGPU deem window/canvas creation out-of-scope, it's likely a similar solution to the winit Rust crate is needed for WASI.

Goals

MVP

The MVP of this proposal should include a way to create windows/canvases and receive input events for them. It should be easy to polyfill for the Web and implementable in engines via e.g. the winit crate. The window handle should be usable in the WASI WebGPU proposal.

Post-MVP

Include querying of window system capabilities that are supported, including:

Setting a window's position
Setting a window's size
Setting a window's displayed name
Setting a window's displayed icon
Capture the Cursor
Enter fullscreen mode

And functions to do these actions.

Discussion

How to deliver the Events to the application? Control flow should not remain in the application, because that conflicts with implementation on the Web.
- Should there be a required exported function by the application that gets called for each event?
- Is there a function reference type in the component model, so you can specify an event handler?
- Or should it require an async main function and wait for async support in the component model?
Are there any more important features missing?
Should this proposal be included in WASI WebGPU?

sunfishcode commented 4 months ago

I don't have experience with Windowing APIs, but I can answer some things here:

* How to deliver the Events to the application? Control flow should not remain in the application, because that conflicts with implementation on the Web.

  * Should there be a required exported function by the application that gets called for each event?

Yes, I recommend starting with this. It's simple, it can be implemented with current features, and it's probably good enough to get quite far.

  * Is there a function reference type in the component model, so you can specify an event handler?
  * Or should it require an async main function and wait for async support in the component model?

Another option might be to model incoming events as a stream<event>, when the stream type is added. But if people are interested in windowing now, we should build something that works with the features we have now; we can adjust in the future as new features arrive.

* Should this proposal be included in WASI WebGPU?

The first thing to do would be to write an initial Wit interface, and if possible, the next is to prototype it, similar to the wasi-webgpu prototype. That way, it'll be easier to answer questions about how this should relate to webgpu and other APIs.

tareksander commented 4 months ago

I created a repository now and added an in-progress WIT definition. How can I test the definitions in wasmtime? Are there tools to generate host bindings?

sunfishcode commented 4 months ago

Cool! And yes, for some examples see the example-runtime in the wasi-webgpu prototype, or the hello-embedded prototype.

tareksander commented 4 months ago

I made a first working prototype now. I had a problem with cargo component though: If I want to be able to run the app as a CLI app in wasmtime, it has to include wasi:cli, but if I include that, that conflicts with the probably builtin definition of the run function in the wasm32-wasi toolchain. I made a copy World for now that doesn't include wasi:cli and load the component as one that does include it, and that works.

I think I'll use event data provided by winit to flesh out the events, and many useful events are still missing. All in all though, I thought it was harder to implement a WASI proposal, even such a small one like this. I think I can set the MVP to be a bit more ambitious now.

Test instructions:

cd example-apps/example1
cargo component build --release
cd ../../example-runtime
cargo run example1

MendyBerger commented 4 months ago

Hi 👋, wasi-webgpu champion here. Would love to discuss how we can work together. What's your preferred method of communication? Are you on the bytecode alliance zulip?

tareksander commented 4 months ago

Are you on the bytecode alliance zulip?

I am now.

What's your preferred method of communication?

I'm more familiar with Matrix, but Zulip seems similar.

WebAssembly / WASI