Swim Issue on HoloLens2

[powellsky]

Steps to reproduce the issue using Unity and Hololens2

• Get checkered paper
• Create a cube with size 1x1cm (0.01 scale in Unity)
• Attach ObjectManipulator to the cube (I also recommend adding RotationAxisConstraint to block rotation on all axis)
• Re-position the cube where the paper is and match the cube with one of 1x1cm squares
• Look at the cube from different angles (good practice is to look from above and walk around it) and try to spot any misplacement differences

We are currently developing a micro scale application for HoloLens2 where millimeters matter. We have noticed a swim effect described here. From the guideline, first we made sure that HoloLens2 detects eye calibration, using the code from Assets/MRTK/Examples/Demos/EyeTracking. Next step was to test different reprojection methods. I did not spot any difference between Auto Planar and Depth Reprojection method. We also tested another method to manually set a focus plane for the object at every frame (with shared depth buffer):

        var normal = -Camera.main.transform.forward;
        var position = transform.position;
        XRSubsystemHelpers.DisplaySubsystem.SetFocusPlane(position, normal, Vector3.zero);

Unfortunately, there was still visible swim effect. Using similar steps to reproduce the issue, I also tried to use built in Microsoft app for 3D objects showcase, where I did place smallest squared objects I could in the area of paper lines. I immediately spotted exact same behavior. Swim effect can go up to 1cm depending on the distance we are looking. I understand that from comfort guidance you shouldn't look at object from closer distance than 40cm because of vergence-accommodation conflict, although this doesn't seem to me like a referring to this issue.

Am I doing something wrong here? I will be extremely grateful for any feedback/advices.

Unity editor version

2020.3.0f1

Mixed Reality Toolkit release version

2.7.0

[polar-kev]

Hey @powellsky, this appears to be a platform limitation--a combination of the tracking and optical systems. I will share this internally to see if there are any other opinions from other teams.

[powellsky]

I will add a note on that as we are struggling to find solution. We found out that this might be a micro-parallax effect explained by Stanford Computational Imaging Group. They introduced a solution to it via gaze-contingent stereo rendering technique. We are wondering what techniques are used by HoloLens 2 to overcome this issue? Also, if there is any way to manually access those functionalities and modify it for our needs. We managed to use Device Portal API for IPD retrieval/overwrite, although we noticed that IPD is being set dynamically by underlying software, which makes perfect sense as IPD is gaze dependent, but the underlaying software prevents us from customizing any values. Please correct me if we are going in the wrong direction on this one.

[rolandsmeenk]

To improve things a bit I recommend using WorldLockingTools: https://github.com/microsoft/MixedReality-WorldLockingTools-Unity.

[powellsky]

To improve things a bit I recommend using WorldLockingTools: https://github.com/microsoft/MixedReality-WorldLockingTools-Unity.

Thank you for your suggestion. We are already using it. World Locking Context is in place, along with Adjustment frame, so everything is world locked.

[rolandsmeenk]

Ok, good, and did you also apply pinning known reference points? Drifting occurs due to "the scale problem" and it may help to add multiple spacepins with known real world distances. Typically this is advertised for fixing larger scale drifting, but it may also help with smaller scale drifting although there still will remain errors due to the nature of inside-out tracking.

[powellsky]

@rolandsmeenk That was very informative. I tried using QR codes as known reference points. Each QR was statically positioned not far from my object of interest, I also added to my QR prefab; SpacePin.cs script provided by WorldLockingTools. Another method I tried was making my object of interest a child of space pinned QR object. Please correct me if I was doing something wrong, otherwise, could you tell me please which method (first or second) would be more appropriate to address this issue?

[stale[bot]]

This issue has been marked as stale by an automated process because it has not had any recent activity. It will be automatically closed in 30 days if no further activity occurs. If this is still an issue please add a new comment with more recent details and repro steps.

[IssueSyncBot]

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visithttps://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!