Depth Occlusion issue with Level Travel

[Junnn424]

Hi all, Hi @MartinSherburn, Thank you for the response on this page, moving my issue here and added some more details: We found that there are 2 behaviors with Depth Occlusion after level travel (or reopen the same map),

1. If we have occlusion ON (no matter it's hard or soft occlusion): Depth Occlusion works perfect at the first level, but once we travel to another map or use "Open level' to open the same map again, it starts to render differently on each eye (see pic reference below) recording: https://youtu.be/by_Wx87PESo

2. If we turn occlusion ON and then turn it OFF before reopen the level, the Occlusion renders the same on both eyes but the position of it is offset from the physical object: https://youtu.be/jOAhBRKNaAQ

More details: At game start, I start at "Map_Lobby_01" using BP_VRPawn, the BP_VRPawn already has a passthrough layer (reconstructed/ underlay) attach to its camera. When I press "Oculus X", it cycles through "Hard occlusion"/ "Soft occlusion"/ "No occlusion" . Then, I use "Open Map_Lobby_01" to reopen the same map again. In the first recording I was in Vanilla Unreal 5.4, so there was no soft occlusion.

Things I tried:

• Switch to oculus branch UE and tested soft occlusion, soft occlusion works, depth issue after level travel persists: https://youtu.be/Kofk7k68qJ8
• Remove the passthrough layer on pawn and add it at begin play same as in occlusion sample, as show below, the problem persists:

[MartinSherburn]

Thanks for the report and detailed information @Junnn424, I'm going to try replicate this in the OcclusionSample and see if we can figure out what is going on.

[MartinSherburn]

@Junnn424 I've tried to replicate what you described in the Occlusion sample. What I did is that I added a new input action so that when the B or Y button is pressed it calls Open Level to reload the current level. When I did this I wasn't able to reproduce the problems you described (I'm using the Meta fork of the engine). Cycling through the occlusion modes and reloading the level doesn't cause any issues. Here is what I added to the level BP:

I'm wondering if there is anything else specific to your project that could be causing this issue. If you are able to reproduce the issue in the Occlusion sample that would be really help to debug this.

[Junnn424]

hi @MartinSherburn Thank you for getting back to me and share your testing! I did found the reason at the end, please try add "Reset orientation and Position" at begin play in Occlusion sample project, and I believe you would see the same behavior. FYI I put it under Occlusions map at begin play: I guess opening the map might not be an issue then, it's just because I had the reset node at begin play.

I should be able to not use that node in our project, but I believe that's a pretty commonly used node and hopefully we could still use it in other project. Thank you for checking!

[MartinSherburn]

@Junnn424 great, I was able to reproduce it with the "Reset orientation and Position" node!

I found the problem and have a fix for this. It will be in v71. But in the mean time you can apply the patch locally to unblock yourself. The changes are fairly small, see below:

diff --git a/OculusXR/Source/OculusXRHMD/Private/OculusXRHMD.cpp b/OculusXR/Source/OculusXRHMD/Private/OculusXRHMD.cpp
--- a/OculusXR/Source/OculusXRHMD/Private/OculusXRHMD.cpp
+++ b/OculusXR/Source/OculusXRHMD/Private/OculusXRHMD.cpp
@@ -4417,6 +4417,7 @@

        SwapchainIndex = DepthFrameDesc[0].SwapchainIndex;
        const float WorldToMetersScale = Frame_RenderThread->WorldToMetersScale;
+       const auto InverseBaseOrientation = Settings_RenderThread->BaseOrientation.Inverse();

        if (DepthViewProj != nullptr)
        {
@@ -4428,7 +4429,7 @@
                DepthFrustum.zFar = DepthFrameDesc[i].FarZ * WorldToMetersScale;
                FMatrix DepthProjectionMatrix = ToFMatrix(ovrpMatrix4f_Projection(DepthFrustum, true));

-               auto DepthOrientation = Frame_RenderThread->TrackingToWorld.GetRotation() * ToFQuat(DepthFrameDesc[i].CreatePose.Orientation);
+               auto DepthOrientation = Frame_RenderThread->TrackingToWorld.GetRotation() * InverseBaseOrientation * ToFQuat(DepthFrameDesc[i].CreatePose.Orientation);

                // NOTE: This matrix is the same as applied in SetupViewFrustum in SceneView.cpp
                auto ViewMatrix = DepthOrientation.Inverse().ToMatrix() * FMatrix(FPlane(0, 0, 1, 0), FPlane(1, 0, 0, 0), FPlane(0, 1, 0, 0), FPlane(0, 0, 0, 1));
@@ -4488,7 +4489,7 @@

            // 2. The headset may have moved in between capturing the environment depth and rendering the frame. We
            //    can only account for rotation of the headset, not translation.
-           auto DepthOrientation = ToFQuat(DepthFrameDesc[i].CreatePose.Orientation);
+           auto DepthOrientation = InverseBaseOrientation * ToFQuat(DepthFrameDesc[i].CreatePose.Orientation);
            if (!DepthOrientation.IsNormalized())
            {
                UE_LOG(LogHMD, Error, TEXT("DepthOrientation is not normalized %f %f %f %f"), DepthOrientation.X, DepthOrientation.Y, DepthOrientation.Z, DepthOrientation.W);

I hope that helps.

[Junnn424]

Thank you @MartinSherburn , it works just fine with reset orientation and position now!

[MartinSherburn]

Ok great! Thanks a lot for spending the time to report, find the repro case and test the fix. Really appreciate it. I'm going to close this task down now.