Camera mounted components in r360 don't work in VR mode

[amitrajput1992]

Description

After migrating react-vr projects to react-360, camera mounted components don't work as expected in VR mode. I was able to mount components manually to the camera object, and they follow the camera rotation in mobile mode, but gets stuck on the screen at a location in VR mode.

screenshots:

Expected behavior

Camera mounted components move with the camera rotation in VR mode.

Actual behavior

Camera mounted components get stuck at a position on the canvas in VR mode.

Reproduction

Since I could not find a straight forward way to mount components onto the camera as was in react-vr's mountComponent, mounted the components manually using:

const node = r360.compositor.getCamera();
  return r360.runtime.context.createRootView(name, initialProps, node);

This seems to work for both desktop and mobile mode but not in VR mode. What I could figure from debugging a bit was, in VR mode, the mono camera's quaternion is not updated and displays the same rotation values. My understanding is, since these rotation values are not updated on the camera (only the leftCamera and rightCamera is updated from the VRFrameData), it continues to look at the direction it was previously looking at and any components mounted on it seem to look in the same direction.

Also, I tried to update the camera quaternion manually in the frame loop which seem to work, but then the reticle raycaster exhibit wierd behaviour by getting fired multiple times on intersection with any element. (onEnter and onExit being fired continuously)

Accomplished the above with following code:

const camera = r360.compositor.getCamera();
camera.position.set(r360._cameraPosition[0], r360._cameraPosition[1], r360._cameraPosition[2]);
camera.quaternion.set(r360._cameraQuat[0], r360._cameraQuat[1], r360._cameraQuat[2], r360._cameraQuat[3]);
camera.updateMatrixWorld(true);

Additional Information

• 'react-360' package version: 1.0.0
• 'react-360-web' package version: 1.0.2
• Operating System: Linux mint
• Browser: Oculus Browser (Samsung gear VR, Oculus GO), Chrome: 68.0.3440.106
• VR Device: Google Cardboard, Samsung gear VR, Oculus GO.

[andrewimm]

This is the recommended way to attach components to the camera in React 360: https://github.com/facebook/react-360/tree/master/Samples/HeadlockedSurfaces

[amitrajput1992]

@andrewimm quick question, it seems that we are not adding the mesh as a child to threejs camera with react-360 but updating the rotation angles for an object added to the scene with the change in camera's quaternion. If my understanding is correct, wouldn't there be a bit of lag and it would seem like the component is being dragged along the scene? The example you mentioned works best for a simple reticle, but for a heavier mesh such as a Fuse or a loading indicator, wouldn't the render performance suffer?