Add Mode that Does Not Require WebGL

[frastlin]

Hello, I would like to question the requirement for a WebGL layer in a WebXR session. WebGL is assuming the application will have some kind of visual rendering. There are plenty of examples where XR does not require visual rendering. Currently audio and visual modes are the most advanced XR modalities, but touch, taste, and smell XR sessions will soon become robust enough to handle their own XR sessions. This topic and this topic in particular give examples of nonvisual use cases for XR.

I would like to see a 3rd mode be added that goes along with immersive and inline. This mode would only give access to the sensors and does operations for world tracking and everything WebXR does that is not related to WebGL. This 3rd session would be a universal basic mode used for all nonvisual experiences, or could be used if someone wants to create their own visual viewer rather than using the tools in the other XR modes, such as with web GPU.

[RealJoshue108]

@frastlin There will be a need for co-ordination between 'modes'. How would this be managed to ensure any non-visual or other abstraction of an environment is in harmony with other modes. Or do you see XR sessions being cherry picked based on a users preference for one over another?

I'm happy to hear more about your 'third mode' idea. Are you thinking something like an XR virtual cursor/buffer type interaction?

[Maksims]

I can definitely see a possible AR application with DOM rendering, with positioning elements using CSS transforms. Simple text overlays over anchored objects using DOM is a valid path too.

[frastlin]

@RealJoshue108 I see a simple API that gives access to values needed for visual and audio positioning, but with no actual GL layer created. I think it would be ideal if users could cherry pick what modalities they want to view, but this issue is focused on developers cherry picking modalities they want to use. One use case is a wayfinding application that uses inertial odometry. Neither visual or auditory modes need to be present, but the user's position is needed. One option for audio feedback in this setup is determining a path the user needs to take, and playing a sound in front of the user in spacial audio. This sound could move several meters ahead of the user as the user moves around the map. Another option could be a movie watching application where your head needs to be pointed in a particular direction for the movie to be active. if you move your head, the movie stops. You don't need webGL for this, you can use the native video element, or some other method of rendering the video.

[RealJoshue108]

@frastlin +1 to this idea of modal 'picking' or muting, which has bandwidth and rendering reduction overhead options also. Again the issue of synchronising between different modes would need to be discussed.

[Manishearth]

I'm very wary of doing something like this until we understand the space of actual devices that work this way. Ultimately, WebXR is constrained by what the devices allow, and a lot of devices require visual frames to be submitted. So the best path forward here is typically to submit blank frames, which content can do as well.

If there are eventually XR devices that are not visual-based it would be good to understand their extent before designing a mode that targets them specifically.

@RealJoshue108 I see a simple API that gives access to values needed for visual and audio positioning, but with no actual GL layer created.

This exists as the device orientation API, fwiw.

[frastlin]

Does the device orientation API have the option to access XRSpace?

[Manishearth]

Does the device orientation API have the option to access XRSpace?

Nope, it's a completely different API.

I do think the WebXR API should be able to support cases where no WebGL content is required, but ultimately this API supports device capabilities and we don't have devices that are explicitly nonvisual. Since visual devices need frames submitted to them anyway the approximation of "content can submit a blank frame each time" is close enough and not much is gained by adding further API surface to paper over it.

[frastlin]

Do smartphones need a frame submitted to them?

[Manishearth]

Do smartphones need a frame submitted to them?

I don't think so.

[cabanier]

FYI if the page use WebXR Layers, it can set it up so there are no layers and thus no need for WebGL.