Default gestures for non-Index controllers to facilitate sign language

[InconsolableCellist]

Is your feature request related to a problem? Please describe.

Gestures are used for non-finger-tracked controllers like the Quest series of controllers, and making certain hand shapes is critical for users to use sign language.

Describe the solution you'd like

There are a number of gestures that currently work as expected for the standard system (VR ASL), but some that don't.

The ones that are necessary to support are:

• Hand open (thumbstick, trigger, and grip all let go)
• Victory (thumbstick hold, trigger let go, grip hold)
• Finger point (Th hold, Tr let go, G press)
• Hand gun (Th let go, Tr let go, G press)
• Thumbs up (Th let go, Tr press, G press)
• Fist (Th hold, Tr press, G press)
• Rock n Roll (Th hold, Tr press, Gr let go)

I've included a chart in Additional Context. This may be easier to explain in person or in a video, and feel free to contact me.

Describe alternatives you've considered

I suspect that the gestures can be modified using the tools available in Resonite, but these should definitely need to work by default to allow people that use sign to communicate with users that know it. There's a large deaf and hard-of-hearing community in VRChat that would find sign extremely difficult to use in Resonite as it currently stands, by default.

Additional Context

[The-furf-of-July]

This could probably be developed into a fairly easy-to-install system if the right tools are available/implemented.

I'm unsure about making it available by default, though. Changing how all gestures work by default is liable to break things. Tools like the mute helper serve similar function, but are not needed by all users. Having a system made and put in a location that is easy for new users to access might suffice.

[InconsolableCellist]

I think it'd be worth trying to modify the default gestures to at least do "victory" and and "rock n roll." The rest are either working or close, from my testing tonight. The devs could see what breaks and decide from there. A more advanced system isn't really necessary for basic functionality, though there are lots of options for more advanced gestures, similar to VRC-ASL_Gestures.

What could break doing this? It'll affect what your armature does on the hand bones, but the controller inputs won't change, so presumably things dependent on them won't change?

[The-furf-of-July]

On further consideration, the main things that would break would be the admittedly janky gesture systems we already have that use bone positions to track gestures on Index. Though, it may cause more issues than before on avatars with poorly or unusually structured hand bones. But that was already an avatar issue to begin with.

That just leaves the remaining difficulty, which would be the playerbase having to adjust to gestures. If it's only a couple, then it might not be that bad. But going from what we have now to specific hand signs rather than loose positions would likely be jarring. So the factors to consider would be players already used to the current look and feel, and maaybe avatars with the "wrong" number of fingers.

[shiftyscales]

Presently, hand posing is handled via skeletal pose data fed into Resonite from the VR runtime (SteamVR, Oculus). We do not directly handle hand posing at present- but rather drive the rotations of fingers using that skeletal data via the Hand Poser component. @InconsolableCellist @The-furf-of-July.

To that end, there are additional hardware and software options that can improve finger fidelity a bit, e.g. Vive Hand tracking which is available to SteamVR headsets in-app via a setting in the settings menu for no cost, or there are hardware options like the Leap Motion- although we have to upgrade our integration to use the latest version as per #8.

(Based on prior testimony I've heard regarding these options, however, they may not be fully up to the task of tracking signs reliably.)

Accessibility is an important topic to me, and I'd wish for us to be able to support and accommodate everyone's needs on the platform as best we are able to.

To that end- do you have any additional reference material available for how signing is handled on other platforms? E.g. which gestures are mapped to which characters?

I am familiar with that platform's hand gestures- but not how they are used in signing/how it translates.

For the moment- the best immediate solution for this issue would likely to be the creation of accessibility tools on-platform that could perhaps be accessed through the "Resonite Essentials" folder.

[InconsolableCellist]

@shiftyscales I'm glad you care about accessibility in Resonite! That's great to hear. I made a little video that demonstrates it better than I can explain it, with VR ASL at the beginning that shows how these gestures combine to sign naturally:

https://github.com/Yellow-Dog-Man/Resonite-Issues/assets/23345188/9bb23df6-f3f1-4dfe-b1a0-c93dc9ab3726

I'd consider this the standard set of gestures necessary for VR ASL, and they correspond to the image I sent above. Seven gestures in total, which depend on the grip, trigger, and thumb-presence detector.

[Frooxius]

We'd probably start with a component that allows you to "simulate" hand poses, by pre-defining a set of them in a number of ways and then selecting with an index, to interpolate.

This would serve as a building block to build arbitrary systems on top of it and give a lot of flexibility in how you want to approach this.

[shiftyscales]

Had a chance to raise this issue internally today in a meeting, @InconsolableCellist. The proposal/concept Frooxius posted above was a result of this.

I've created and added an accessibility label to this issue so it, and other related accessibility issues can be tracked in the longer term.

No ETA on when it will be looked at, but when it is worked on, we will let you know/update this issue.

Thank you for your patience in the interim.

[Beethoven-n]

just tested this with someone. there is no V pose, not even a rock and roll gesture. this is very frustrating, people shouldn't have to have the knuckles controllers to sign