search

microsoft / MixedRealityToolkit-Unity

This repository is for the legacy Mixed Reality Toolkit (MRTK) v2. For the latest version of the MRTK please visit https://github.com/MixedRealityToolkit/MixedRealityToolkit-Unity
https://aka.ms/mrtkdocs
MIT License
6k stars 2.12k forks source link

Help with articulated hand inputs. #10407

Closed HyperLethalVector closed 9 months ago

HyperLethalVector commented 2 years ago

Unity editor version

2020.3.24

Mixed Reality Toolkit release version

2.7.3

Overview

Hi everyone!

For Experimentation purposes, I've created a virtual hand within the unity environment, and have been aiming to input the virtual hand joint positions as an articulated hand into the MRTK. I've done so based on the current MRTK Leap Motion Input Provider.

Expected behavior

So far, I've been able to create an Input Provider, and successfully raise/close the input source. I've also verified that the poses for the joints are updating correctly within the Input Provider, with the correct sources for each 'joint' to the MRTK. I would've expected everything MRTK to 'just work'

Actual behavior

However, I'm not seeing any cursors coming out of the hands (even though all my pointers are attached to the articulated hand), nor are any of my virtual 'pinch' gestures. Only the head gaze seems to be working.

Is there something I'm possibly missing?

The codebase for this provider can be found here:

https://drive.google.com/drive/folders/1a81DBVNaS7W6gwd2PYVMI14nKvczHJNc?usp=sharing

Basically, the AutoHand MRTK Skeleton obtains the transforms from the virtual hands, which are then used by the provider.

Steps to reproduce

Grab the above mentioned link, and drop it into the MRTK, you can assign the transforms of the autohand driver.

Any help would be massively appreciated :(

HyperLethalVector commented 2 years ago

I've verified that the 'pinch' interactions and everything else is detected.

On inpsection, the pointer poses aren't getting updated.

RogPodge commented 2 years ago

Hey there,

Sorry for the late response. I believe the reason you aren't seeing any pointers associated with your new articulated hand is because you are missing the appropriate attribute in above the class:

Your implementation: image

LeapMotion Articulated hand definition: image

The reason this is important is because pointers are configured to be available on certain controller types: image

and based on the controller mappings, the AutohandArticulatedHand wasn't detected as an ArticulatedHandController image

Moving the AutohandArticulatedHand into it's own file and adding the following attribute seemed to fix this issue after a project restart, I'd suggest doing that to resolve this issue.

[MixedRealityController( SupportedControllerType.ArticulatedHand, new[] { Handedness.Left, Handedness.Right })]

Thanks for using MRTK! In v3 we should have a much simpler process for all of this :)

HyperLethalVector commented 2 years ago

Words cannot describe how hard I facepalmed after reading this, but I'm over-the-moon that it works very well.

Would there be an interest in doing a PR with this additional virtual hand driver?

RogPodge commented 2 years ago

I think I'd want to know a bit more about how the functionality that the new hand provides. If the improvements/new features it adds are useful, we'll gladly take a look at a PR, review it, and hopefully try to get it in!

HyperLethalVector commented 2 years ago

For sure!

The aim was to allow us to rapidly prototype MRTK interactivity using VR controlled hands, but to do so in an abstract enough way to allow compatibility with any virtual hand system. This is mostly us skirting around not having enough hand-tracking enabled devices for our developers.

IssueSyncBot commented 9 months ago

We appreciate your feedback and thank you for reporting this issue.

Microsoft Mixed Reality Toolkit version 2 (MRTK2) is currently in limited support. This means that Microsoft is only fixing high priority security issues. Unfortunately, this issue does not meet the necessary priority and will be closed. If you strongly feel that this issue deserves more attention, please open a new issue and explain why it is important.

Microsoft recommends that all new HoloLens 2 Unity applications use MRTK3 instead of MRTK2.

Please note that MRTK3 was released in August 2023. It features an all-new architecture for developing rich mixed reality experiences and has a minimum requirement of Unity 2021.3 LTS. For more information about MRTK3, please visit https://www.mixedrealitytoolkit.org.

Thank you for your continued support of the Mixed Reality Toolkit!