High level scene graph manipulation API

[jbouvier-onshape]

Should this model tag allow access to an API to manipulate or interact with a scene graph - for example, selection of a part in the model, providing metadata stored with the selected item ? For example in a gilt format, we could select a node and emit an event containing metadata with the name of the node. As well we could set basic feature to the selected item, as visible, hover feature.

[bhouston]

Basically something akin to WebGL / Canvas? I think that is a big ask at this time.

I think this proposal is akin to the initial tag and the Canvas and WebGL APIs came later.

[robertlong]

Agreed, that we should have some high level API for interacting with model elements.

Until recently I was working on a project called Third Room. It's an open source virtual world platform built on Matrix. A large part of this platform was the ability to write scripts that can modify a glTF scene graph in a world. We called this API the WebSceneGraph API.

WebSceneGraph is a high level API that mostly mirrors the data structures of glTF. After working with USD for the last month, I think it could work very well with USD as well. Rather than submitting draw calls with an API like WebGL or WebGPU, WebSceneGraph is focused on modifying scene graph data structures.

A simple script looks like this:

world.onload = () => {
  const directionalLight = world.findLightByName("DirectionalLight");

  world.onupdate = (dt, time) => {
    directionalLight.color[0] = (Math.sin(time) + 1) / 2;
  };
};

In Third Room, worlds run untrusted code that modify the world's scene graph. To accomplish this in a browser we run WebSceneGraph scripts in WebAssembly. As such the API is defined first and foremost as a WebAssembly API. We then embedded QuickJS in the WebAssembly context to provide a JavaScript API. Developers are free to write scripts in any language that compiles to WebAssembly or in JS. While this property of sandboxing content may not be useful when used with the model element, it does mean there is an avenue to providing a WASM API for model element scripts.

The above script can also be written in C like so:

#include <math.h>
#include "websg.h"

Light *directionalLight;

export void websg_load() {
  directionalLight = websg_world_find_light_by_name("DirectionalLight");
}

export void websg_update(float_t dt, float_t time) {
  websg_light_set_color_element(directionalLight, 0, (sin(time) + 1.0) / 2.0)
}

WebSceneGraph also contains a very lightweight ECS system. Its entities are nodes in the scene graph, in our case these are the glTF nodes. Components have explicit data types and can be deserialized from the source content (glTF or possibly USD). We started working on a glTF extension for defining these components and attaching data to nodes. Apple also has an extension for doing this in USD for RealityKit. Systems are not explicitly defined and you're responsible for determining how to schedule and run them. This is intended to keep things flexible for developers who may or may not want to write strictly ECS code. There is a built in entity query API though, which can be used to find nodes with the specified components.

The API also has support for transforms, materials, lights, basic physics, mesh data (including dynamic meshes and instanced meshes), a basic 2D UI API, and more.

Our intentions were to submit this API for consideration by W3C once we had fully implemented an example of it in Third Room. However, the team was recently laid off and we weren't able to complete this work. Currently, Third Room has a mostly working implementation of the API. We have a tutorial where you create a basketball game using the JavaScript API. We've put a lot of thought into the design of it and it could be a decent starting point or at least an example of what a model element API might look like.

I'd love to talk more about adding a high level API to model element. Please let me know if this is of interest to the group.

[gmazzucchelli]

/facetoface