Black Environment From Cube Camera in VR

[drbct]

Description

Ahoy!

We ran into what appears to be a bug with regards to CubeCamera in VR. Specifically, calling CubeCamera.update and then pointing Scene.environment at the texture of the underlying WebGLCubeRenderTarget works fine while rendering regularly, but produces completely black reflections in VR.

In the interest of quick reproducibility, the live example linked below was built around the WebXR API Emulator extension. However, we observed the same behavior testing with two actual headsets (Quest 2 and 3 using the Meta Quest Browser).

Reproduction steps

1. In the live example, press Space to recalculate the scene's environment map using a CubeCamera which is located just in front of the metallic cube you're looking at.
2. Use A and D to move the red plane behind you. Press Space again to verify that the reflections are being updated.
3. Enter VR.
4. Press Space. The cube should turn black.
5. Exit VR.
6. Press Space. The cube should remain black.

Code

The offending code appears to be in lines 73 through 75.

scene.environment = scene.background;
cubeCamera.update(renderer, scene);
scene.environment = cubeRenderTarget.texture;

Here, we're first resetting the environment to the initial background. In the live example, this comes from a RoomEnvironment, but all of the above applies equally to a cube map loaded from individual images. Then, the cube camera renders the scene and the resulting cube texture is set as the scene's environment.

We also tried pointing the envMap property of the cube's material to cubeRenderTarget.texture and obtained the same result.

It appears as though the cube map is still being rendered correctly in VR. This can be verified by changing line 75 to

scene.background = cubeRenderTarget.texture;

and performing the first update in VR. Subsequent updates produce a feedback loop, but that is to be expected. This suggests that the problem lies in applying the texture to Scene.environment while VR is active.

Live example

• jsfiddle-dev

Screenshots

No response

Version

r162

Device

Headset

Browser

No response

OS

No response

[Mugen87]

@drbct Can you please replace all materials in your scene from MeshStandardMaterial to MeshPhongMaterial and test again? Scene.environment can't be used in this context so you have to assign the environment map to the envMap property instead. But since you said using envMap produces the same (wrong) result, it should be good for a test.

I want to clarify if PMREMGenerator breaks in XR. In might be necessary to disable XR in PMREMGenerator during the generation process. The generator is not used with MeshPhongMaterial.

[drbct]

Can confirm. This example using envMap

• is broken with MeshStandardMaterial: https://jsfiddle.net/7hdo4bsj/1/
• works with MeshPhongMaterial: https://jsfiddle.net/7hdo4bsj/2/

[Mugen87]

Can you test with one of the build files of https://github.com/Mugen87/three.js/commit/d2b06eeb0ef72b54ddc0113e7bf30ba208aa2a6f if the issue is fixed?

I've added a patch to PMREMGenerator that hopefully fixes its usage in XR.

[drbct]

Works like a charm; Thanks for the quick response!

[fromtheghost]

Apologies if I'm misunderstanding this fix, but I'm not able to get the THREE.CubeCamera to update properly while using WebXR, and it seems to be related to the pmremVersion.

Here's a sandbox: https://codesandbox.io/p/sandbox/pmrem-threejs-fgcmdc

It's a slightly modified version of the official dynamic cube reflection example.

I've added a VR button. It renders fine on desktop. In my Meta Quest 2, it renders, but the perspective is warped and unusable.

The problem can be fixed by commenting out the MeshStandardMaterial in line 49 and uncommenting the MeshBasicMaterial in line 55, and then uncommenting line 113:

cubeCamera.renderTarget.texture.pmremVersion--;

With that, it renders fine on the Meta Quest, seemingly because the PMREM update is bypassed.

Is anyone else experiencing this? Or is using the cubeCamera to generate a dynamic envMap not supported yet in WebXR?

[Mugen87]

I can't test fiddles and codesanbox apps with a Quest since VR is not allowed inside their iFrames.

Any chances to share a standalone test application that demonstrates the issue?

@drbct Do you see any regressions in r163 with your XR apps compared to the dev version we have tested with? Is the original issue of this thread still solved with the current release?

[fromtheghost]

It should work on your Quest if you open in a new tab that uses SSL: https://fgcmdc.csb.app/

[Mugen87]

Thanks! Indeed, I can confirm the perspective is completely broken.

Would it possible for you to make a test with this version of the build file? Does the bug disappear?

https://rawcdn.githack.com/mrdoob/three.js/d2b06eeb0ef72b54ddc0113e7bf30ba208aa2a6f/build/three.module.js

[fromtheghost]

Thanks for the quick reply. Yes, I'm still seeing the distortion with that version loaded: https://yvqwqv.csb.app/

Here's the sandbox link if you want to check it over: https://codesandbox.io/p/sandbox/pmrem-threejs-forked-yvqwqv

[Mugen87]

Okay, so it seems it isn't a regression within r163. I need a closer look for this one...

[Mugen87]

I can't edit the sandbox so I can't share a new link. But I suggest you remove the lines:

sphere.position.z -= 75;
cube.position.z -= 75;
torus.position.z -= 75;

You then have to move the camera away from the origin:

camera.position.z = 75;

The cube camera should be position at the location of the sphere like so:

cubeCamera.position.copy( sphere.position );

I was wondering why OrbitControls is broken in your app and the root cause is positioning the camera at the origin. I believe the side effects we see in XR could be related to the same issue.

Besides, a XR scene should honor real world scale so it's better to position the like so:

camera.position.set( 0, 1.75, 2 );

This requires a different transformation of the sphere, torus and box as well.

[Mugen87]

If the issue still persists, please file a new issue. The issue we see is unrelated to the problem originally discussed here.

[fromtheghost]

OK - thanks for helping. I'll investigate further.