Forearm point tracking

[Naphier]

Describe the problem

The wrist node is helpful for hand tracking, but often times we also need to understand the forearm's position. Leap Motion implements this which allows for objects to be anchored to points along the forearm (Leap.Hand.Arm.ElbowPosition -- which is used to create a vector from the wrist to elbow). HL2 seems to do this with the Start Menu button, but there appears no clear/simple way to do this with the SolverHandler. We'll probably end up hacking something together but an estimation of the forearm's vector or the elbow point would be helpful.

Describe the solution you'd like

Add Forearm to the trackable hand joints for SolverHandler. The actual tracking point could be at the tip of the forearm where it connects to the wrist. Would also be helpful to have an elbow estimation.

Thanks!

[Naphier]

@keveleigh Our attempts at hacking this are fruitless. Just not enough info. Can you tell me if / how we can get this on the roadmap? Use cases include: robotic arm control (can only do like 5-dof with an IK and no elbow/forearm info), avatar control (if we have an estimation of the forearm then we can IK chain up to the shoulders with fairly good reliability), menus attached FLAT to the forearm (provides the user with something to press against for those with depth perception issues (me!) -- I realize we can probably do on the back of hand and palm, but there's very limited space there). Let me know if there's some better place for us to make an appeal for this sort of feature. We really need it. Thanks!

[keveleigh]

@CDiaz-MS is working on Leap Motion support. She can comment on potential plans for exposing additional joints supported by that platform.

As far as getting forearm / elbow tracking from HoloLens 2, I'm not sure the best way to provide that as far as MRTK is concerned. I believe the start menu button location is based on an offset from the wrist joints.

There are some potential examples of arm-based menus, though they're under our Menu positions that are not recommended guidelines.

[Naphier]

Thanks Kurtis. To be clear, we don't desire to use the Leap Motion with the HL2 we just want elbow approximation or vector of the forearm. Understood about the menus. We have a forearm one working much better than what we see there (suspend the placement of the tracked menu when overlap occurs basically -- works well because ppl don't tend to move both arms simultaneously when touching one hand to an arm -- think smartwatch usage, pretty common and not all that uncomfortable for quick actions ;)).

[Naphier]

Some further validation that we need some forearm reference instead of just wrist rotation: https://www.forbes.com/sites/jessedamiani/2020/03/30/yur-launches-yurwatch-a-virtual-smart-watch-for-tracking-fitness-metrics-in-vr/#48981c7062c8 https://www.youtube.com/watch?v=grvYL547fps While it may not be a good idea to put interactables on someone's forearm, it would be highly valuable to be able to add graphics to the user's forearm in ways that yur is demonstrating here with VR handtracking.

[derekfreed]

I like the idea of having that for displaying info as you said, but maybe not interactions since that would likely interfere with the HL start menu

[Naphier]

As long as your not on the inside of the wrist then this isn't a problem, really. But it is too bad that MS decided they're the only ones that can use this area. We've been doing forearm mounted controls in apps with Leap for years now ;)

[Zee2]

The HoloLens 2 platform does not track forearm joints, unfortunately. @Naphier the HoloLens's system shell does not consume an extra forearm joint to display the start button; it is, in fact, just extrapolated from the palm orientation/vector. I would suggest using a similar heuristic to create forearm-mounted UX controls, if necessary.

[Naphier]

@Zee2 similar heuristics don't work for our use case as we wanted the elements flat against the arm. This is a feature request. Are you denying it by closing it? There are many use cases as pointed out above and it is a shame that this is being so easily discarded.

[keveleigh]

@Naphier For something like a new forearm tracking point, we can't really add that in MRTK ourselves as we can only use the data provided by the platform (more info at https://docs.microsoft.com/windows/mixed-reality/develop/native/hands-and-motion-controllers-in-directx#articulated-hand-tracking). I'd suggest filing a Feedback Hub feature request to get it in the hands of the feature team.

Specifically, for "flat against the arm", is the wrist joint (with some offset) stable enough for that use case?

[Naphier]

No. We explored that. Because the Home button is on the underside of the arm it is a bit in the way. And if you use the back of the arm the wrist flexes a lot more so the elements lift off the arm too easily. We settled on the back of the hand and it works OK but not ideal. I'm fairly certain we put this in Feedback Hub but got nothing back. It's a moot point now.

On Wed, Aug 4, 2021 at 4:20 PM Kurtis @.***> wrote:

@Naphier https://github.com/Naphier For something like a new forearm tracking point, we can't really add that in MRTK ourselves as we can only use the data provided by the platform. I'd suggest filing a Feedback Hub feature request to get it in the hands of the feature team.

Specifically, for "flat against the arm", is the wrist joint (with some offset) stable enough for that use case?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/MixedRealityToolkit-Unity/issues/7414#issuecomment-892947280, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACNJYPFRKSEVKDEREMBZ7FDT3GOJPANCNFSM4K4KAM7Q .

--

Sean Mann NaplandGames.com http://naplandgames.com/ facebook.com/NaplandGames