`renderOptions={{ whenVisibleOnly: true }}` breaks Meta's "Immersive Web Emulator" extension

[sevaseva]

Setting renderOptions={{ whenVisibleOnly: true }} property on <Engine> component "breaks" Meta's "Immersive Web Emulator" extension [1].

More specifically, the scene initially renders as expected in the non-immersive mode in Chrome desktop browser. The button for entering the immersive mode does render as expected, and, when clicked, the immersive mode seems to activate, however, in the emulated immersive mode

EXPECTED: the viewport of the browser renders the immersive view of the scene (with two VR controllers initially in front of the camera, by default, etc)

OBSERVED: the scene isn't rendered at all, instead the viewport is all white. a 2D [exit] button is visible at the bottom right (normally it's not visible).

To be clear: removing renderOptions={{ whenVisibleOnly: true }} property, restores the expected functionality.

---

I reproduce it with

scene.createDefaultXRExperienceAsync({
    uiOptions: {
        sessionMode: "immersive-vr",
        referenceSpaceType: "local-floor"
    },
    floorMeshes: [scene.getMeshByName("...")],
    optionalFeatures: true,
});

Update: Chromium v 121.0.6167.160, stable channel, Linux, Gnome, Wayland Immersive Web Emulator extension v 1.5.0

though I assume/hope that most any "immersive-vr" session should reproduce this (and therefore am being lazy and not offering a more specific minimal repro).

---

Any idea what may be involved, @brianzinn ?

I thought it may or may not be a "bug" in this library or that extension, but I though it's worth investigating and trying to fix.

If it helps, I can create a minimal repro kit and file an issue with the extension and see what Meta folks say.

---

[1] The extension: https://github.com/meta-quest/immersive-web-emulator https://chromewebstore.google.com/detail/immersive-web-emulator/cgffilbpcibhmcfbgggfhfolhkfbhmik

[sevaseva]

To be clear, this reproduces only in the extension; does not reproduce in Meta's or Apple's headsets.

Also, react-babylonjs v 3.1.26

[brianzinn]

The only difference really, is that the render isn't being called when the canvas itself is out of view. So, there is an observer on the canvas. It would seem to me that once you enter the immersive mode that potentially the canvas is being altered by the emulator?

[sevaseva]

Possibly. I didn't look at the extension's code yet.

[brianzinn]

Have a look at this example: https://[brianzinn.github.io/react-babylonjs/examples/basic/pause-renderer/](https://brianzinn.github.io/react-babylonjs/examples/basic/pause-renderer/)

You could copy the Intersection Observer code from here: https://github.com/brianzinn/react-babylonjs/blob/master/packages/react-babylonjs/src/Engine.tsx#L17

There's not a lot there and it has the same net effect as the Pause Renderer sample code. Then in your XR helper you can observe when you exit/enter your 'immersive-vr' session. https://doc.babylonjs.com/features/featuresDeepDive/webXR/webXRExperienceHelpers

Those should be all the things you need to tie into in order to work around what looks like an emulator specific issue. To be fair it's a lot of extra coding to work around the emulator. I am not aware of a static way to tie into that observable though, but otherwise I could add a shim to this library by looking at the XR details and hooking into enter/exit. Maybe I could add a callback where the xrHelper is provided and then I can hook into those additional details. Just trying to think out loud for possible solutions.

[sevaseva]

I confirm the emulator extension does mess with the canvas: in the process of starting the immersive session the canvas becomes invisible before it becomes visible again. I cannot 100% confirm at the moment that the canvas does not temporary become invisible in the process of entering the immersive session in hardware VR headsets (don't have a headset next to me for a couple more days), but it seems like either it does not (and therefore rendering loop does not stop even if renderOptions.whenVisibleOnly === true) OR somehow pause/resume of rendering does not affect things in a hardware headset.

---

Let me see if I understand what you propose me to try: You propose that I stop using the renderOptions.whenVisibleOnly option and instead create and activate my own IntersectionObserver that would set/unset my own boolean piece of state (name it myOwnIsPausedState) and I would propagate that as a prop to the react-babylonjs Engine component like <Engine isPaused={myOwnIsPausedState} .... My custom IntersectionObserver would set myOwnIsPausedState to false when the canvas is invisible (to effectively replicate what renderOptions.whenVisibleOnly is doing) ... but it will also ensure that myOwnIsPausedState is never false (always true) if an XR/immersive session is active. Is that right?

---

Assuming I got that right:

I can see how that would ensure that rendering never stops in 'immersive-vr' session, but something tells me that the emulator might make the canvas invisible before it activates the immersive session... I am not sure about that yet thought. I could try that, I suppose.

But I am also thinking of simply setting renderOptions.whenVisibleOnly = true in production and to false elsewhere. That's a much smaller change and I don't really care a) whether the emulator works in production and b) about battery/performance drain (from rendering into invisible canvas) in non-production environments.

... In fact I mostly care about the emulator to work when I am debugging locally, so I can (and already do) even more simply comment out the renderOptions.whenVisibleOnly = true line locally, successfully debug, and never commit that change).

[sevaseva]

And talking about working around this incompatibility in this library,

I am not aware of a static way to tie into that observable though, but otherwise I could add a shim to this library by looking at the XR details and hooking into enter/exit. Maybe I could add a callback where the xrHelper is provided and then I can hook into those additional details. Just trying to think out loud for possible solutions.

I am assuming you are thinking about ensuring the rendering is never stopped when a immersive session is active, is that right? Maybe we first need to establish if that's the issue here.

[brianzinn]

The babylon team just merged an XR PR with fixes - they indicated it may resolve the issue: https://github.com/BabylonJS/Babylon.js/pull/14753/files

It will be on version 6.44.0 - let's wait for that version and then if it's not resolved I can spend some more time here. Cheers.

[sevaseva]

Looks like an epic PR! I will test and report back after 6.44 releases.

---

I see that Raanan mentioned that some issue related to some emulator. I'm curious what issue he refers to; we will find out, I guess. This issue never reproduced in plain BJS apps (unless someone conditionally stops the rendering loop when the canvas isn't visible using IntersectionObserver or another technique, I guess) or in apps that use this library unless whenVisibleOnly: true is in use. Nevertheless, no reason not to try on the other side of 6.44.

---

In other news, I do clearly see that the canvas briefly becomes invisible between click on the "enter vr immersive session" button and the immersive session starting (starting successfully if we never stop the rendering loop) under the emulator extension.

I am 90% sure that [canvas briefly becoming invisible] does not happen in the process of starting the immersive session on an actual headset. (will be able to verify that 100% in a day or so)

When I confirm that it is the emulator that makes the canvas briefly invisible (as it moves it within the DOM, under a different parent div node that emulator creates), I will hope that making it invisible isn't essential for what the emulator needs to do. And I will file an issue with the emulator to see if they can stop doing that.

[sevaseva]

nah, that's some other issue... I'll retest with 6.44 anyway though

[brianzinn]

I didn’t see anything in the PR that would fix this issue. I’ll wait for that version though to see and otherwise propose what I need to work around the issue.

[brianzinn]

thanks @sevaseva. your PR is deployed with version 3.1.29.

[sevaseva]

Thanks Brian