search

Malcolmnixon / GodotReadyPlayerMeAvatar

Godot ReadyPlayerMe Avatar Reader
MIT License
9 stars 2 forks source link

VR integration? #8

Open Eisenspalter opened 5 months ago

Eisenspalter commented 5 months ago

I really like the possibility to use Ready Play Me Avatars in Godot that is great! But I'm totally confused. I thought I could use Ready Player Me avatars in my VR project with this plugin. But the demo project is not a VR project. Is that possible to use it in VR or not? And if so, is there a demo project for VR integration?

Malcolmnixon commented 5 months ago

This library loads the avatar and configures it to be driven by an XRBodyTracker and XRFaceTracker.

The demo project uses the GodotXRVmcTracker and is intended to be used with XR Animator as this provides full-body and face tracking suitable for populating the XRBodyTracker and XRFaceTracker data.

Most XR Headsets only provide hand-tracking, and sometimes upper-body tracking. Only the Quest Pro does full body and face tracking - and only then when you use their OpenXR headers as they haven't released their extension definitions as part of the official Khronos spec and public releases yet.

In order to support avatar-driving on more primitive headsets; "something" needs to be able to synthesize the XRBodyTracker and XRFaceTracker data from the limited input data available - IK, Gait, and Audio-to-Face synthesis. I've been toying around with some ideas for the face using ML models based on audio samples; but the IK/Gait modeling is probably going to be some custom IK stuff.

If you have a Meta headset like a Quest 3 or Quest Pro you could play with the https://github.com/GodotVR/godot_openxr_vendors demo project (you would need to compile the extension yourself) and that demonstrates avatars driven by the Facebook tracker data. It wouldn't take much effort to add this asset and configure the RpmSettings to use the default OpenXR trackers rather than the VMC ones in this demo.