Research on NoClip OOT3D Depth

Background

XRPackage implements a WebXR runtime that allows multiple applications to be composited together.

This works using WebXR and taking identical poses for each app and each frame (view and projection matrix) and interleaving the resultant WebGL rendering commands into the same WebGL

Additionally, alpha zero in the fragment shader means that nothing is rendered and the color does not write to the virtual WebXR

Note that XRPackage controls the top level render loop here, meaning that we control the aspects of frame timing, positioning, and so on, so WebXR can be controlled from desktop mode.

NoClip is a site that implements many game engines to render their environments correctly on modern hardware. Here we are focused on Ocarina of Time 3D, "OOT3D", which we would like running as an XRPackage that correctly depth composites with other applications.

Goal

The goal is to modify a fork of NoClip oot3d so that when it is packaged and loaded into XRPackage, the resultant app composites depth-correct with other content.

The rendering engine for NoCLip oot3d WebXR currently has two problems we need to address before we can get its depth correctly into the world:

First, NoClip renders in two passess and the final pass is basically a plane in front of the headset. That means that from the view of XRPackage, the only thing we are getting from NoClip is a "screen" rendered in front of the face. Naturally, this does not draw correct depth even if we somehow had it. Which leads to the second issue:

Second, NoClip does not keep depth up to the part where it draws the screen plane. So we either need to get rid of this pass or thread the depth info through to it, and properly draw/depth test it in.

Third, NoClip does not track depth with the same math as THREE.js. That is, its math is either inverted (going from 1 to 0) near to far rather than THREE.js's 0 to 1 near to far, or it is using different near/far settings. I am not sure which is the case. This would need to be fixed in the WebGL rendering command set up to have it match the THREE.js setting before the depth will composite correctly.

Fourth, NoClip seems to render a skybox that clears out the alpha to 1, meaning it draws overtop of the scene and the content that was there. We'll need to not draw the skybox, or use alpha zero to make sure that it doesn't interfere with other rendering.