Web GUI extension

[geraintluff]

I'm looking to build CLAP plugins as WASM bundles, where all the CLAP API calls are exported from the WASM module. This looks (so far) like it would just work, which would be absolutely awesome.

One piece of the puzzle I haven't figured out the UI. So far, my effects specify a page to open inside an <iframe>, and that web-page posts state updates which are translated into custom (non-CLAP) function calls.

Option 1: dedicated extension

I would love to have an official CLAP extension to:

1. specify the path for a web-view UI
2. post/receive messages from that web-view (clap_ostream_t again?)
3. get notified about close/open

Option 2: webview API type for clap.gui

This would mean adding a clap_webview *webview member to the clap_window union, and a CLAP_WINDOW_API_... constant.

The clap_webview would need to contain a (host-provided) method for posting messages, and registering a (plugin-provided) listener for incoming messages. On the other hand, all the scaling/sizing stuff is already in clap.gui.

---

With either of these options, the plugin's clap.gui implementation (for native GUIs) could set up the webview and then translate messages to/from the other API.

[baconpaul]

I think a moderately opinionated option 2 is a good choice

I wonder if the approach is to add a header called gui_webview.h which doesnt contain a host or plugin extension but instead defines the struct clap_web view and the gui constant and then use the gui extension with that object (which as you suggest probably contains two things which look like event queues and two things which look like streams)

my gut is that the easier it is for a clap gui to advertise the web view in a way which is consistent with the gui extension the easier it will be for devs which is why I sht away from a totally separate extension.

But I haven’t written a web plugin yet so this is just thoughts not answers

[geraintluff]

I also reckon option 2 would be neater. What's the process for trying to move this forward - writing a strawman gui_webview.h?

[abique]

I'm skeptical that the webview is the right direction, though it could be tried... There are multiple problems:

• javascript, security and the huge bloat of requirements to compile the engine
• impossible to to keep up to date and follow web standards

I think WASI 0.3 will bring wgu and webcanvas.

But then we'd also need the windowing system events (mouse, keyboard, ...). And it is imaginable that this comes with a later update of WASI, we'd just need to be patient and wait. I imagine that wasm + gui is not ready yet.

On the architecture side, I don't think it is reasonable to expect every host to implement wasm support. wasm support should come from an opensource transparent adapter:

• one native opensource plugin that will transparently host wasm plugins and proxy them to the daw (only the factories needs a proxy I suppose)
• this adapter could have its own extensions, and I'm thinking one that can give you the mouse, keyboard, ... events; one that can give you the wgpu context.

To me the real and first milestone is the adapter plugin + headless wasm plugins.

[geraintluff]

@abique I've obviously explained something badly. 😅

Hosts wouldn't need to support WASM, or have a JS engine. WASM would be an alternative target (akin to Windows/Mac/Linux), for in-browser use. The native builds of the plugin would run exactly as normal - internally they'd probably have a thin wrapper setting up a system-provided webview (e.g. using CHOC) and forwarding messages, but that's not the host's concern.

Also, in the browser the WASM runs in a non-GUI context (AudioWorklet) and should primarily be handling audio. That code shouldn't be drawing anything, or handling mouse/keyboard - even in an "adapter" plugin, or future WASI versions, since WASM doesn't support threads. The UI stuff should be happening entirely within the web page's HTML/JS.

Here's a sketch of what I was thinking - I think the rightmost one is the adapter plugin you mentioned, but it's completely optional.

[abique]

Ah I didn't understand your goal.

I'd be in favor of one or many dedicated extension(s) for Web CLAP plugins. If it can fit into a single ext, you can have the file include/clap/ext/draft/web.h otherwise include/clap/ext/draft/web-*.h

[abique]

As for the web gui, I'm not sure how it is supposed to work, but as you said it is about having a frame or a div and then the plugin is autonomous for rendering and dealing with user events.

You could make a single ext just for that web-iframe.h? I imagine there might be more things that you'll want to hook in the future so I believe that having many small exts is the way to go.