search

microsoft / MixedRealityToolkit-Unity

This repository is for the legacy Mixed Reality Toolkit (MRTK) v2. For the latest version of the MRTK please visit https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity
https://aka.ms/mrtkdocs
MIT License
6k stars 2.12k forks source link

Objects leave hand when head moves. #10198

Closed reeseras closed 2 years ago

reeseras commented 3 years ago

Describe the bug

When holding an object with hand tracking, moving your head (and not your hand) will cause the object to drift out of your grip. It's extremely noticeable in small objects. (Is this a problem with hand tracking in general, or is it MRTK?)

To reproduce

Steps to reproduce the behavior:

  1. Create an object in unity about the size of your finger.
  2. Add NearInteractionGrabbable and ObjectManipulator as well as a collider to the object.
  3. Grab the object with your hand, then move you head back and forth.

Expected behavior

The object should stay in place with the hand.

Screenshots

https://user-images.githubusercontent.com/26148056/132071026-82c7bb70-7e0a-4d91-b224-140dce8d9da6.mp4

Your setup (please complete the following information)

Target platform (please complete the following information)

Additional context

Again, this might be a limitation of hand tracking, but just in case I posted it here.

david-c-kline commented 2 years ago

@RogPodge, can you please try a repro on 2.7.2 (or the main branch)?

@reeseras, can you also try updating your project to MRTK 2.7.2?

RogPodge commented 2 years ago

I couldn't get this to repro on 2.7.2. There's some jittering which may be attributed to the natural movement of the hand, but I didn't see any drift to the degree that was shown in the video.

reeseras commented 2 years ago

I'll try updating mrtk as soon as I can; my admin is out for 2 weeks.

reeseras commented 2 years ago

I tried to update using the Mixed Reality Feature Tool found here. I don't have a mixedrealitytoolkit folder so I don't know how to check the version.

The problem persists. Also, I can't see my hands anymore (they work fine in the oculus menus).

There's also a new error: System.IO.DirectoryNotFoundException for ...\Library\PackageCache\com.microsoft.mixedreality.worldlockingtools@8c5f43b0484f-1631137558589\Packages\Microsoft.MixedReality.Unity.FrozenWorld.Engine.1.1.1\lib\unity\iOS-ARM64\FrozenWorldPlugin.framework

RogPodge commented 2 years ago

@davidkline-ms do you know what this frozen world exception is?

reeseras commented 2 years ago

If it helps, I also get this error after I upgrade: image

Removing the file allows me to launch unity.

RogPodge commented 2 years ago

Sorry for the late response, we're juggling some other feature work at the same time as our existing work.

Are you still experiencing this issue? If so, can you provide a package or some means of recreating the scenario shown in the video? We can't seem to reproduce the drift on our end.

reeseras commented 2 years ago

Yes, I can create a package of the project, but how do you want me to get it to you? Github doesn't seem to like the .zip I put it in.

RogPodge commented 2 years ago

A google drive link would be viable.

reeseras commented 2 years ago

I think this will work: https://drive.google.com/file/d/1qyYwB2G8ALn2FyaxtrGW_VjEkDX-6rfX/view?usp=sharing

Go to scenes and run the test scene. There's a green syringe in there. Try to grab it (hands aren't visible). Now move it so you're looking over top of the syringe and move your head side to side (much like the video). The syringe still moves quite a bit even though your hand is not moving.

RogPodge commented 2 years ago

Thanks! I'll try to take another look at this when we have a bit more capacity from our other endeavors.

reeseras commented 2 years ago

Ok, I tried a fresh installation of MRTK on a completely new project. I'm on Unity 2020.3.19f1 with MRTK 2.7. After loading it in I picked up a block and it had the same problem as above. For some reason I'm unable to record it off the computer, so I tried building it to the headset. Unfortunately, building it to the headset gives me a dialogue box requesting that I use controllers. I loaded the scene and it doesn't have any hand tracking at all when building to the headset. I guess that's a whole separate problem, but that's why I don't have a video of it not working.

For reference, this is the video I followed to install MRTK: https://www.youtube.com/watch?v=rb5GZVDREdk

My point is, I still had the same problem in MRTK 2.7.

reeseras commented 2 years ago

Update: I did get MRTK 2.7 to work inside of the project. Everything seems to be working fine but the problem described above still exists/is worse:

https://user-images.githubusercontent.com/26148056/138773081-a3c195f2-6a39-420f-9df3-73bfd1821da9.mp4

It seems to have a lot of trouble moving objects toward/away from the camera.

provencher commented 2 years ago

I'm looking at this bug, and it reminds me of some of the behavior exhibited by the far ray pointers, where the distance from hand to head influences the offset from the grab point. It's designed like this to bring objects closer.

I'm wondering if the object manipulator here is setup in such a way that it's using that behavior for proximity grabs. @reeseras my recommendation is to try and repro this with some of the sample objects in the hand interactions demo scene. If you cannot repro it, try and copy the setup of those components on your syringe.

RogPodge commented 2 years ago

Extremely sorry I had not been able to look at this @reeseras, I got swamped with some high priority asks the week right after I promised to look at your project :(

I will try to get back to you later this week about my findings

RogPodge commented 2 years ago

I think this will work: https://drive.google.com/file/d/1qyYwB2G8ALn2FyaxtrGW_VjEkDX-6rfX/view?usp=sharing

Go to scenes and run the test scene. There's a green syringe in there. Try to grab it (hands aren't visible). Now move it so you're looking over top of the syringe and move your head side to side (much like the video). The syringe still moves quite a bit even though your hand is not moving.

Hey @reeseras @jcellswo I didn't see a test scene in this project, just asset models and scripts.

Feel free to ping me in the holodevelopers slack directly and I can respond and tackle this issue much more quickly. There's also an active broader community of MRTK and Hololens developers on there!

Again super duper sorry for how long it's taken to revisit this.

Holodevelopers slack: https://join.slack.com/t/holodevelopers/shared_invite/zt-5d6fsq82-m3vHQGYBAvJfjtAR6_wBSA My username: @Podge

reeseras commented 2 years ago

Thanks! I'll mention this in Slack, but meanwhile here's a video of the same thing happening in a blank version of MRTK 2.7.2 installed on Unity 2020.3.19f1. I installed it using the Mixed Reality Feature Tool. I didn't change any settings. I duplicated the blue block and rescaled it to make a shape similar to the syringes I was using in the original video. Everything seems to work as normal, so I'm guessing this has existed in MRTK up to this point?

https://user-images.githubusercontent.com/26148056/140178129-84fe041c-550f-476e-ac93-c2b0af2b9535.mp4