Hand Skeletal Pose System

[Frooxius]

Currently Neos uses native skeletal model from the Index, Touch and Vive controllers and from Leap Motion if available.

Windows MR controllers lack a skeletal model resulting in motionless hands.

Adding a gesture source component would help with this scenario, as well with cases where the user wants to make specific pre-made gestures, rather than using their real fingers directly.

How many people are interested in having this component available?

[Jazzneo]

I think alot rift user be happy with it

[apiedev]

Also, if you're using the latest 2019 Input System, this would allow for essentially any supported headset or controller to work in the future.

[Frooxius]

Neos is using it's own input system. That's not really relevant to the gesture system though, this one just deals with a system that allows posing fingers using pre-made poses/animations, rather than live skeletal system.

[apiedev]

Oh, is it not built on the latest Unity XR Input System? That's what I meant. The controllers without skeletal structures could all share similar gestures using the Unity XR input mapping system. Windows MR, Finch controllers, and basically any future controllers/XR devices would just work with NeosVR natively so long as they're supported by the integrated Unity XR input system.

[Frooxius]

No. For the most part Neos is its own engine, with it's own subsystems. Asset management, physics, scripting, scene management, networking and many more.

Like I said, that's not really relevant to this topic, this is about posing the fingers using pre-made poses and animations, not a skeletal model coming from the controller (Neos already supports that for all controllers that have it - Index, Valve, Touch, even Leap Motion).

[apiedev]

I know, I was just trying to give a suggestion since the Unity XR Input system allows you (or even users of Neos) to create their own controller input mapping to their animations if setup correctly. For instance, you can bind things such as turning a controller 90 degree to the right to an animation pose. You could even get a Vive Tracker to do animated poses, even though it doesn't have buttons, with that system.

EDIT: Again, I don't know how the underlying framework of Neos is setup, was just suggesting it might be something to look into. Although, it does require Unity 2018 or 2019 and updating to completely new versions of Unity can be a pain.

[Frooxius]

Ah I see. Thanks for the suggestion! But yeah it can't really be used because Neos has its own input system and handling and this needs to be done on Neos' side. Plus it fits within Neos' philosophy, because it'll let you edit and configure everything in-game.

Neos has a generic hand skeletal input system that it utilizes for the controllers. This proposed gesture system would essentially work with that and feed it skeletal data from a pre-made pose or animation, rather than some finger tracking solution.

Upgrading generally isn't an issue though, Neos is pretty easy to transfer to a new versions, we're currently using the latest stable version 2019.2.2f1

[TehTurk]

Updated the name to reflect the issues solution better.

[StarfishHidari]

Poking this to see what the status of this is, since interest was polled back in 2019 and not much comment since then. Currently still 5th most voted issue

[Toxic-Cookie]

Gonna poke this too. Would be pog to have.

[Abysmal2134]

I'm interested in this too, but I would appreciate if someone would clarify what exactly adding this gesture source component will entail exactly, because I might be under a misconception.

As far as I remember, Frooxius mentioned hand posing in the context of being able to set up grip poses when equipping items (when that system got revamped). So is this Hand Gesture/Pose System going to make it possible to pose the hand's fingers, so that you can equip a matchstick/teacup/etc. and hold it with two precisely posed fingers, or is it relating to something else completely?

Sorry if I'm misinterpreting things, but I really liked the grip pose changes, and really wanted to vote on an issue that would continue improving on those changes (like the mentioned 2 handed wielding of items like spears/rifles, and posing of fingers for smaller and more delicate items), and this is the only issue I could find related to hand posing.

[Frooxius]

This component would let you use an in-world hierarchy as the source of the finger skeletal data for your avatar. You could do anything you'd like with this hierarchy, like animating it and so on. It would essentially act the same way as the skeletal model from SteamVR or Oculus does.

However this is actually less relevant today. Since this issue was created, Windows MR has gotten skeletal model in SteamVR, so the fingers do move now and allow for basic gestures, so the main use would be if you wanted to build your own replacements, without having to bake it into particular avatar.

[StarfishHidari]

This was still wanted back when this issue was originally created because it would be helpful for other controller types too, allowing you to define custom hand shapes based on controller input, similarly to what other platforms allow you to do. With my oculus headset I cannot make a V hand shape at all for example. WMR users weren't the only people wanting this

I would really really like to be able to define custom hand shapes to be able to be used, every other alternative way I've gone about fixing this is very inefficient and clunky.

[Abysmal2134]

So it's essentially a prerequisite for a unified skeletal system for the hands to pose all the different rigs used by other headsets, kind of like we calibrate the avatar's full body tracking to a dummy rig? Thus every avatar's hand would be able to be puppeteered to the component independent of the headset rigs.

Seems like a pretty big undertaking though. Something on the same scale as redoing the FBT. It's at the same time inspiring and daunting to see so many possible new features that can improve Neos, so even if this isn't a priority now I'm glad it's feasible and being considered.

Sorry if I go on a rant with my posts, but Neos makes me passionate, so expect my posts being a bit overzealous when they are on something that ignites my anticipation.

[Frooxius]

No, the skeletal system is already unified, it's how the skeletal model works with the controllers - it remaps that skeletal model to your avatar. This is just adding another source of the skeletal data for the same system, one that you can control.

It will allow for making it independent of the avatar - you could have gesture system that'll work with any avatar that has a proper finger rig.

I'd also add some components to blend/switch between the skeletal sources, so you could use the controller one most of the time and only trigger specific gestures as overrides.