GLSL front-end framework

[kig]

Imagine this: you have a compute shader that animates every element on the page using a fluid simulation. At animation frame time, the latest simulation results are applied to the page elements as CSS transforms. You can still mess and interact with the elements with touch / mouse / keyboard. How do you wire this up?

Collect element coords into a big buffer, pass it over, get another big buffer back, apply results?

[kig]

Client-side web application structure:

• UI sends an event stream to the client.
• UI has fast-path hardcoded meanings for some events (scroll to navigate DOM tree)
• Client can request UI to add timeout & frame events to the event stream.
• Client can also request UI to add other Web API events to the event stream (geolocation, camera access, microphone.)
• Client can request service worker to send messages to the server.
  • Service worker can cache idempotent message replies.
• Server reply stream is passed from service worker to client.
• Client can also request a WebSocket to send and receive a message stream from a server.
• Client generates an outgoing stream of DOM trees that are rendered by the UI.
  • Some elements in the DOM trees can be modified in-place for performance (canvas)
  • Full-on tree rebuilds may be slow, preferred to change only parts of the tree that have changed
• Client can generate audio streams that are sent out-of-band.
• Client can respond to some events in an async manner (e.g. add new content after click), some require synchronous handling (request fullscreen, initiate file download, play initial audio, event.preventDefault)
• DOM trees must be built on the main thread
• Canvases can be updated in workers

Tenets:

• Keep main thread as light as possible
  • Should only pass on events, respond to sync events, and build DOM trees
  • Receives event stream, serializes events to ArrayBuffer, hands it to dispatch worker
  • Receives tree stream, deserializes tree ArrayBuffer to DOM edits, applies them
  • Main thread is realtime
  • Must yield every few milliseconds to browser
  • Incremental event processing
  • Incremental tree builds
  • No long GC pauses
• Workers are black boxes, main thread doesn't need to know how they do their thing
  • Worker can run on CPU or GPU or distributed
• Use offscreen canvases
• Collect events into larger batches, dispatch on frame
• Workers do all network fetches, maybe even including image loading
  • Images rendered into offscreen canvases, decompressed using Wasm libs
• Performance benefit is in getting off the slow main thread and JS/C++ land ASAP and processing everything in GLSL. Even on CPU and quad-core, this can be 10x faster than naive C++. If you're GPU-friendly or have 32 cores, >100x faster is not impossible.

[kig]

Input stream limits:

• Input events max ~4 per frame.
• WebSockets on most browsers choke at 120 MB/s. Exception is Linux Firefox which can sustain 500 MB/s if you tune it right.

In a nutshell:

• Main thread: Async realtime stream processor with transferables and some way to define sync event handlers for some UI events. Incremental processing, batching.
• Workers: Pipelined stream processors.
• Everything is operating on local data, server and client datastores kept in sync. Offline first (= keep data locally cached, have a good sync algo.) Compress & decompress local data-at-rest on the fly with parallel zstd or such.

People would likely use this kind of library only for doing lots of computation or having large local datasets, like semi-local web apps where you load data from local storage. Here maximizing bandwidth and storage efficiency are nice to have.