GGV Z-depth manipulation does not behave correctly correctly if hand rays disabled

[sostel]

Describe the bug

I would like to use any pointer (head gaze, eye gaze, hand rays) to target a hologram on HoloLens 2 and then when tapping down, I would like to start moving it. Ideally I want to reuse the same ManipulationHandler script, but while the movement behavior seems to be working well when using hand rays, it shows some weird depth movement when having hand rays turned off.

To reproduce

Steps to reproduce the behavior:

1. If you use the HandInteractionAllExample.unity scene and start it with the DefaultHololens2ConfigurationProfile, holograms can be targeted and moved using hand rays.

2. In comparison, if you switch the profile to EyeTrackingDemoConfigurationProfile, you can look and air-tap to select a hologram and then while pressing your index finger and thumb together move the hologram. The left/right/up/down movement seems ok, but when I move my hand to and away from my body, the grabbed target moves on an unexpected trajectory.

Expected behavior

I would expect that when I move my hand toward my body that the currently targeted (no matter how I targeted it) is coming toward me.

Your setup (please complete the following information)

• Unity Version 2018.4.14f1
• MRTK Version v2.2.0

Target platform (please complete the following information)

• HoloLens 2

[sostel]

@julenka Do you know who I could ask about this?

[julenka]

Yes, this would be @MenelvagorMilsom and @thalbern. Assigned to @thalbern for triage.

[julenka]

@thalbern I've been thinking to write tests that verify interaction behavior, something like, with hand rays, moving hand 10 cm when object is at distance 1m should move object exactly 0.7m, or somethign like this. Similarly, with not rays but ggv I was thinking we could have similar "moving hand to right 10 cm when object is distance 1m should move object 10 cm to right". That way we could protect against regressions in movement algorithms.

[MenelvagorMilsom]

I had a look at this one on device yesterday. It did feel a bit weird, but I couldn't place why. The object went where I expected, but it was almost as if the rotations were strange. Perhaps related to #6510?

Just a thought, I didn't dig into the code.

[sostel]

@MenelvagorMilsom Did you try it with eye gaze? This is what I observed: I would look at a hologram, tap down and then move my hand straight out from my body. The target would move on a curved line to the side.

[MenelvagorMilsom]

Yeah, eye gaze is what I tried with. I didn't experiment for too long, but it felt to me like the trajectory of the grab point was expected, but because the object rotated about this point it made it trajectory of the object as a whole feel strange.

I'm not sure if the hand rotation should ever cause the manipulated object to rotate when using gaze/gestures interaction, even on HoloLens 2.

A good way to test my theory is to compare the trajectory of objects set to RotateAboutGrabPoint and objects set to RotateAboutObjectCentre. If the trajectory of objects with RotateAboutObjectCentre is unexpected in the same way as it is for RotateAboutGrabPoint then perhaps this is caused by something else.

I may have some time to test this later.

[sostel]

Any updates on this?

[julenka]

This also reproduces in the new ObjectManipulator script which is going to be the preferred / active object manipulation script for MRTK 2.4 and beyond.

[julenka]

This big is a side effect of #5645, which was built to fix #5640. As a temporary workaround, this can be fixed by changing the code that gets the Rotation of GGVPointer to be:

        public virtual Quaternion Rotation
        {
            get
            {
                Vector3 look = Camera.main.transform.forward;
                return Quaternion.LookRotation(look);
            }
        }

The proper fix involves modifying the MoveLogic that ObjectManipulator uses to use a different algorithm than what is used for hand rays and motion controllers.

[julenka]

It looks like the workaround above causes movement up/down and left/right to regress. As an alternative, I would recommend trying the old movement behavior shipped in MRTK 2.0, before it was refactored to support hand rays better. That behavior could work better for ggv manipulations, as it was tuned specifically for ggv.

[stale[bot]]

This issue has been marked as stale by an automated process because it has not had any recent activity. It will be automatically closed in 30 days if no further activity occurs. If this is still an issue please add a new comment with more recent details and repro steps.

[IssueSyncBot]

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!