search

mbucchia / VirtualDesktop-OpenXR

An implementation of the OpenXR standard for Virtual Desktop
MIT License
209 stars 7 forks source link

Hand Tracking? #14

Closed DonFlymoor closed 7 months ago

DonFlymoor commented 7 months ago

Since the Oculus OpenXR version as well as ALVR has hand tracking, I assume that oculus like/oculus imitation hand tracking could easily be implemented, allowing any PC game with hand tracking support to use the quest's hand tracking

mbucchia commented 7 months ago

Virtual Desktop doesn't retrieve hand joints information from the device. So there is nothing I can implement today.

We're definitely in talks to support it, but it's definitely not easy and requires coordination between the Quest app, the Steamer app (both are the more complex part, not in my control) and finally (the easiest part) VDXR.

I think one reason it's not going to be prioritized is because there are practically no apps today implementing OpenXR hand tracking. I've only heard of DCS implementation in 2.9, and I hear it's bad and most people don't use it :/

DonFlymoor commented 7 months ago

I can imagine if it's added a lot of games might add support for it. The quest app already captures hands, although it only uses that for VD interactions - I would imagine sending and receiving hand data could use the same methods the head tracking does, although I'm no expert.

The most ideal situation would be to use multimodaltracing and perhaps emulate the vive controller which tracks finger movement, although this would currently require quest pro controllers (for the future quest 2 controllers are being looked into)

mbucchia commented 7 months ago

I can imagine if it's added a lot of games might add support for it.

OpenXR had support for hand tracking for years, first with Ultraleap and then with Quest (since 2021), and still no games are using it. Virtual Desktop isn't going to move the needle here.

I would imagine sending and receiving hand data could use the same methods the head tracking does, although I'm no expert.

Proper hand tracking requires 26-joint skeletal tracking and a certain degree of motion prediction that has its own properties. This is very different from head tracking. It also requires gesture detection, which Meta did not implement as cross-vendor API (and therefore app developers cannot use it in a generic way).

perhaps emulate the vive controller which tracks finger movement

I assume you mean "Index controller", and this is what we are talking about already (it's the same OpenXR API for hand joint tracking and Index finger sensing). Nonetheless, it doesn't change that nearly no applications currently uses this API. The two (big) OpenXR applications with Index controller support today are Bonelab and Pavlov VR, both will refuse to use "plain" hand tracking because they also require buttons and joysticks, that your hands do not have.

mbucchia commented 7 months ago

Closing as not actionable for now.