Using the Depth API without OVRCamera prefab

[vtimejc]

Hi,

We have managed to get the depth API in and working, but as we don't use the OVRCamera prefab we get some problems.

Specifically the "holes" appear offset when the camera rig is not at the origin. Setting _customTrackingSpaceTransform didn't work. We worked around this by directly applying our own eye poses in a derivative of EnvironmentDepthTextureProvider.

We then have the problem that reprojection latency causes the occlusion to "wobble" as the head moves.

Looking at the code, there seems to be a workaround for that in the sampling - but that doesn't work for us, as we can't match camera poses to the age of the depth buffer. In theory we could if we could use the createTime field in EnvironmentDepthFrameDesc, but firstly we don't know what the scale is and (more importantly) it is always zero...

[TudorJude]

Hey,

The customTrackingSpaceTransform assumes you have an OVRCameraRig, so changing it won't do anything. Can you share your modified code?

[vtimejc]

I can describe it. I derived a class from EnvironmentDepthTextureProvider, say CustomDepthTextureProvider

Inside that I have:

    private void Update()
    {
        thisFrame = (thisFrame + 1) & (MAX_FRAMES_LATENCY - 1);
        eyePoses[thisFrame].left.position = leftAnchor.transform.position;
        eyePoses[thisFrame].left.rotation = leftAnchor.transform.rotation;
        eyePoses[thisFrame].right.position = rightAnchor.transform.position;
        eyePoses[thisFrame].right.rotation = rightAnchor.transform.rotation;

        TryFetchDepthTexture();
        if (!isDepthTextureAvailable)
           return;

        int sampleFrame = (thisFrame - DepthSampleFrameLatency) & (MAX_FRAMES_LATENCY - 1);
        var leftPose = eyePoses[sampleFrame].left;
        var rightPose = eyePoses[sampleFrame].right;

        var leftEyeData = Utils.GetEnvironmentDepthFrameDesc(0);
        var rightEyeData = Utils.GetEnvironmentDepthFrameDesc(1);

        // These values seem to be valid no matter the clip planes
        float depthNearZ = 0.1f;
        float depthFarZ = System.Single.PositiveInfinity;

        // Calculate Environment Depth Camera parameters
        Vector4 depthZBufferParams = ComputeNdcToLinearDepthParameters(depthNearZ, depthFarZ);
        Shader.SetGlobalVector(ZBufferParamsID, depthZBufferParams);

        // Calculate 6DOF reprojection matrices
        reprojectionMatrices[0] = CalculateReprojection(leftEyeData, leftPose);
        reprojectionMatrices[1] = CalculateReprojection(rightEyeData, rightPose);

        Shader.SetGlobalMatrixArray(ReprojectionMatricesID, reprojectionMatrices);
    }

and

    static Matrix4x4 CalculateReprojection(Utils.EnvironmentDepthFrameDesc frameDesc, Pose eyePose)
    {
        #if !UNITY_EDITOR
        float left = frameDesc.fovLeftAngle;
        float right = frameDesc.fovRightAngle;
        float bottom = frameDesc.fovDownAngle;
        float top = frameDesc.fovTopAngle;
        float near = frameDesc.nearZ;
        float far = frameDesc.farZ;
        #else
        //  Hardcoded values from Quest 3

        float left = 1.376382f;
        float right = 0.8390996f;
        float bottom = 1.428148f;
        float top = 0.9656888f;
        float near = 0.1f;
        float far = Single.PositiveInfinity;
        #endif

        float x = 2.0F / (right + left);
        float y = 2.0F / (top + bottom);
        float a = (right - left) / (right + left);
        float b = (top - bottom) / (top + bottom);
        float c;
        float d;
        if (float.IsInfinity(far))
        {
            c = -1.0F;
            d = -2.0f * near;
        }
        else
        {
            c = -(far + near) / (far - near);
            d = -(2.0F * far * near) / (far - near);
        }
        float e = -1.0F;
        Matrix4x4 m = new Matrix4x4
        {
            m00 = x, m01 = 0, m02 = a, m03 = 0,
            m10 = 0, m11 = y, m12 = b, m13 = 0,
            m20 = 0, m21 = 0, m22 = c, m23 = d,
            m30 = 0, m31 = 0, m32 = e, m33 = 0
        };

        var viewMatrix = Matrix4x4.TRS(eyePose.position, eyePose.rotation, ScalingVector3).inverse;

        return m * viewMatrix;
    }

[kyrylokuzyk-m]

Hi @vtimejc,

Thanks for opening the thread! Can you please answer a couple of additional questions so we can debug the issue faster?

1. You mentioned that you don't use the OVRCameraRig.prefab. Can you please share how do you update the camera pose? Do you use the OVRCameraRig.cs or do you have a custom component for that?

2. If you have a custom script that updates the camera pose, then camera's parent would be your 'tracking space' transform. Can you please try to assign the camera's parent transform to the _customTrackingSpaceTransform?

3. The compensation you're trying to achieve is not necessary and is already covered by the CalculateReprojection() method. That method already calculates the difference between the current depth camera pose and the pose at which the depth texture was captured.

[TudorJude]

Closing issue due to inactivity.