WebXR not supported on WebGPURenderer

[hybridherbst]

Description

As it seems that work on WebGLRenderer will go into maintenance mode to make more space for WebGPURenderer, it would be great to see an initial implementation of WebXR so we can start testing WebGPU as well.

We'd like to test and help, but I'm not sure how WebXR fits into the current architecture of the new system with two backends. I'd already be happy if WebGPURenderer ({forceWebGL: true}) had WebXR support (so, not actually WebGPU).

Reproduction steps

1. Use WebGPURenderer
2. Try to use XR
3. Note that APIs don't exist anymore (getSession(), getCamera(), isPresenting, ...)

Version

r167

Device

Headset, Phones

OS

Windows, Android

[Mugen87]

As it seems that work on WebGLRenderer will essentially stop to make more space for WebGPURenderer,

Sorry, I need to correct this statement since it isn't right. We do not stop the work at WebGLRenderer but just limit the scope. So there will be bug fixes and smaller new features but no groundbreaking refactoring or enhancements.

[hybridherbst]

No problem, adjusted the wording to "will go into maintenance mode", which is what I meant.

[hybridherbst]

From my understanding, there is no specification yet for how to use WebGPU with WebXR (@toji, if you want to chime in that would be cool!).

But what could likely be done is to add WebXR support for new WebGPURenderer({forceWebGL: true}), which would unlock testing and verification of the new backend structure for applications.

@mrdoob do you have an opinion on how WebXR should be changed/added to the backend(s)? I'd be interested in exploring that, and I believe so would @rcabanier. I think it would be great to get WebXR added to the WebGL backend for now, and then later once the specification is ready the WebGPU backend can get support as well.

[CodyJasonBennett]

This needs https://github.com/immersive-web/WebXR-WebGPU-Binding as there is no way to bind to the compositor from WebGPU, nor is it a good idea to mix WebGL and WebGPU on the same page, performance ramifications of a full copy aside. Also not the easiest with coordinate conventions differing, although not too bad with only two views or projections to correct -- quick solution here https://twitter.com/Cody_J_Bennett/status/1658786889577496579.

It would be nice if this wasn't so internal to the WebGL backend (like it is now with WebGLRenderer) as that makes it hard to maintain for texture code-paths specifically, and locks it down to future improvements like multi-view without a rather involved refactor, partially due to limitations or issues on Quest which may resolve in time (or be dropped for alternatives like old-school tricks using instancing/multi-draw or storage memory if you don't have a high number of indices -- remember indexed drawing is an important optimization for TBDR). There are other things like multi-pass (e.g. userland shadows, upsampling) or post-processing which are locked down by Meta which are endemic to a vertical system like this. Reminder I have a $1k bounty on this or #26160 since this is a massive uplift for WebXR as a platform.

[hybridherbst]

@CodyJasonBennett I understand that WebXR on WebGPU isn't even fully specified at this point.

My proposal (https://github.com/mrdoob/three.js/issues/28968#issuecomment-2250907600) is that the WebGL2 backend of the new WebGPURenderer could support WebXR today, and thus open the path towards actually using the "new" three, including the Nodes architecture.

For the time being, XR applications would then use new WebGPURenderer({forceWebGL: true}) to stay in WebGL2.

[toji]

The proposal @CodyJasonBennett linked is indeed the missing piece needed on the browser level to make this work. The good news on that front is that I'm scheduled to start work on that very soon! I'll keep you updated as progress is made.