Support for Quest Multimodal tracking

cerealbowlsystem commented 5 days ago

Is your feature request related to a problem? Please describe.

In version 20.11.0 ALVR added Multimodal tracking support, allowing fingers to be tracked while holding the controllers (both finger curl/splay and controller buttons work at the same time) or for just hand position to be tracked when one or both hands are too far from the controllers. I personally don't know how this is achieved (ALVR devs surely do), but from what I can see SteamVR sees 2 pairs of "controllers" and at any time only two "controllers" are active (green icon instead of grey). I believe the active controllers are simply the ones responsible for hand position and rotation, while the skeleton is always being controlled by the hands regardless. When one/both controllers are far from a hand, that controller's icon becomes grey and the corresponding "controller" on the other pair gets activated, its icon becoming green.

As of writing this issue, this already almost works with Resonite; my fingers move while I'm holding my controllers and the buttons work as well. However, the moment where a controller<->hands switch happens, the corresponding Resonite hand stops moving, both in position and rotation, and it remains like this until the game is closed, The position might also just jump to a very far away place after that happens, perhaps when another switch happens on that hand. Controller buttons keep working tho.

Describe the solution you'd like

The solution might be to have Resonite accept multiple controllers at startup and constantly get the active controller's position/rotation from SteamVR. Multimodal tracking and on-the-fly controller switching already works on VRC, so I believe it's possible.

Describe alternatives you've considered

The only current alternative is to just not use multimodal tracking, or to just rely on the Quest/ALVR's older hand tracking auto switching. This doesn't give us simultaneous controller buttons with finger tracking, tho.