XR_EXT_hand_tracking joint positions different from SteamVR

[oneMillionWorlds]

Describe the bug

I'm finding that the bone poses as reported by VDXR are different from those obtained when running under SteamVR (reported using XR_EXT_hand_tracking extension).

E.g. here is the hand in XDVR

And here it is in Steam VR

It's not that different, but its different enough that the hands don't look right. Here they are with the bone positions rendered

VDXR:

Stream VR:

Environment

• Application/Game and version: Tamarin Test Bed: https://github.com/oneMillionWorlds/TamarinTestBed
• Device: Quest 2
• OS: Windows 10
• Version of Virtual Desktop: 1.30.3 beta
• Extension being exercised: XR_EXT_hand_tracking

To Reproduce

1. Run TamarinTestBed
2. Look at hands
3. See odd positioning

Expected behaviour

Expected that SteamVR and VDXR would produce the same bone positions

OpenXR.log

[mbucchia-msft]

Not really sure what I am looking at here. SteamVR doesn't support "real hands" AFAICT, so this must be while holding a controller?

VDXR "hand joints simulation" is pretty cheap, and I don't really plan on improving it at this time. This is really meant for when a real hand is being tracked, the "hand joints simulation" code is based on Valve's own sample code here to provide a best-effort fallback for apps that rely on fake hand joints when holding a controller. I unfortunately don't have the time nor expertise to provide a better simulation model.

[mbucchia-msft]

The relevant code calling Valve's code is here: https://github.com/mbucchia/VirtualDesktop-OpenXR/blob/0d2a45704aa847f62d819cdbc546237b7c235ccb/virtualdesktop-openxr/hand_tracking.cpp#L305

[oneMillionWorlds]

Yes, this is while holding the controller. They aren't physically accurate (its the best guess based on touch points on the controller) but they look physically plausible from SteamVR. It's interesting that its up to the runtime to figure out the hand positions, I had always assumed (clearly incorrectly) that the device provided its option on them.

[mbucchia-msft]

There might be some simple tweaking in the code I pointed above to make it better. If you feel like experimenting.

I will try a couple of adjustments later if I find some time.

[oneMillionWorlds]

If its going to be cheap and cheerful less might be more. Tamarin itself has a hand joints simulation that it uses when XR_EXT_hand_tracking isn't available. It uses only the grip strength and uses that to open and close the hands. It's simultaneously better and worse; better in that the hands always look physically plausible, worse in that certain poses like pointing the index finger are impossible.

Tamarin grip based bone simulation. It works just by having a number of recorded way points at different grip strengths and interpolating between them.

Although arguably that's going down a dead end.

No worries if its not on the roadmap though.

[mbucchia]

Please note - I added support for XR_EXT_hand_tracking_data_source to manually activate when to retrieve simulated hand joints. This will no longer be done by default, unless the user selects "Emulate index controller".

[mbucchia]

I don't plan on making more changes to this feature.