Framebuffer API

[olanod]

What do you think of my comment in https://github.com/WebAssembly/WASI/issues/171#issuecomment-566061984. Feasible? Not expert in the subject so I might not see lots of small details.

... I'd like to see a frame buffer API some time, for embedded contexts for example a buffer that I fill with some pixel information(e.g. a micro writing B&W pixels of a small e-ink display) is all that would be needed I guess. The frame buffer can be a wasi file descriptor, perhaps the user can request/create more than one buffer or is given one by the host, then with existing APIs something like window resizing could be detected by listening to events on the file descriptor, a __WASI_EVENTTYPE_FD_READ could signal that the buffer changed(window resized?) a __WASI_EVENTTYPE_FD_WRITE could be the requestAnimationFrame of WASI world. Once the user renders pixels on the buffer by hand or with a graphics API like webGPU then the host decides what to do with that, could be a headless set up where the frame-buffer is dumped to a file or telling the compositor of the platform hey here are some pixels!, put it in the window you gave me!(window was created by the run-time host, not the user?).

[programmerjake]

I think having a method to access a memory-mapped frame buffer (not necessarily in GPU memory, can just be system memory) is a good idea -- it exists in all of (non-exhaustive list) Win32, X11, Wayland, KMS/DRM (Linux and BSDs), and nearly all embedded devices with a display and can be easily emulated on other systems.

This allows for fast software rendering when desired, including on GPU-less devices.

This is not a replacement for GPU APIs such as WebGL or WebGPU, which we should also support (except where there is no GPU, such as small embedded systems), but each method should get exclusive access to each Vulkan Surface (or equivalent) to avoid needing complex synchronization.

[pierogitus]

Perhaps the baseline should be a compositor/animation API. You create a surface then animate it with its handle and a matrix (similar to CSS transform). Animating could be a capability right distinct from read/write. Maybe you could even generate a new handle where the read/write capabilities were constrained to a 2D region of the surface.

Also any graphics API brings up the need for an accessibility API

[torch2424]

Hello!

I was thinking about picking this up as a personal project in my free time 😄

@MarkMcCaskey and I gave a brief overview of a similar API Wasmer is currently working on at the December 19th, 2019 WASI Subgroup meeting.

@sunfishcode brought up some good points that a standardized version of this API would not want to use a filesystem, as there may be some instances that there is no filesystem that the runtime is using. But instead, use memory and syscalls to store and draw individual pixels.

@programmerjake

This is not a replacement for GPU APIs such as WebGL or WebGPU, which we should also support (except where there is no GPU, such as small embedded systems), but each method should get exclusive access to each Vulkan Surface (or equivalent) to avoid needing complex synchronization.

Definitely agreed here, I think we should definitely support this down the line, and should make sure this smaller / lower-level API is not perceived as an alternative for them 👍

Also any graphics API brings up the need for an accessibility API

This is a very good point, I did not consider this. We may want to add some accessibility hooks or something of the sort.

---

But just so we are all on the same page. I was imagining this to be a set of syscalls that enabled a low level per pixel drawing. Similar to this articl, that explains this in the context of a linux framebuffer. 😄

If everyone here agrees this is a good starting point for the Framebuffer API, I'll go ahead and start looking for resources on how the proposal and stuff should be done. Thanks!

[torch2424]

Hello!

So we discussed this again at the latest (I think?) WASI meeting on January 16th, 2020.

Unfortunately, the meeting notes weren't posted, but I took these very vauge action items :joy:

• Write witx (or C header API) files, open a PR.

• Keep in mind audio and input.

• Framebuffer probably wont be in wasi libc, probably in a lib-framebuffer.

So I'm going to start tacking this tonight-ish :smile: I'll try and figure things out, and send you a pr @sunfishcode :+1:

[AldaronLau]

@torch2424

Keep in mind audio and input.

What kind of ideas or APIs have been developed so far for audio?

[torch2424]

@AldaronLau None that I know of so far :smile: But in general, it would be a good idea to make sure we don't block the availability of an audio buffer that a WASI host could support :smile:

[unmellow]

does it sound like a good idea to support vector frame buffers? e.g. a frame buffer of vector images as opposed to raster? (as an additional API TBC)

[niutech]

Wasmer I/O Devices provide framebuffer API for WebAssembly, see announcement. Can't you standardize it looking at this implementation?

[MendyBerger]

We're now working on this at https://github.com/WebAssembly/wasi-webgpu/blob/main/wit/frame-buffer.wit