Support facial tracking with Virtual Desktop

[orange3134]

Is your feature request related to a problem? Please describe.

No problem, but this is an enhancement request.

Describe the solution you'd like

Virtual Desktop has the ability to send Quest Pro's facial tracking data. I would like to be able to use this feature for facial tracking in Resonite.

Describe alternatives you've considered

Use the VDFaceTracking MOD created by Zeitheron. https://github.com/Zeitheron/VDFaceTracking/tree/master

Additional Context

No response

Requesters

orange3134

[Frooxius]

Is there any documentation for the Virtual Desktop sharing this data?

It seems like it uses memory mapped files from the mod, but I can't find any documentation for this.

[shiftyscales]

Annoyingly I couldn't find any direct mention/documentation of how Virtual Desktop exposes/passes through the facial tracking data.

The closest I found was a VRCFT module to expose it via OSC from VRCFT. https://docs.vrcft.io/docs/hardware/quest-pro#virtual-desktop-setup https://docs.vrcft.io/docs/hardware/quest-pro#modules https://github.com/guygodin/VirtualDesktop.VRCFaceTracking

So clearly there must be some way that it could be more directly passed from Virtual Desktop to other applications- I just can't immediately find it.

Although VRCFT OSC is already covered by #1843 as a separate issue.

[Frooxius]

I can work with the memory mapped file, but I was just wondering if there's actual documentation for it, rather than just looking at the code of existing implementations.

[shiftyscales]

I'll continue to dig around and see what I can discover.

[Frooxius]

Ok thanks! If there's nothing, that's fine, I can work with one of the implementations and make implementation off that, I was more curious if somebody was already aware of where this is documented.

[webhead2oo9]

Hey there. I grabbed this from the pins on the discord for you. Forward Data needs to be enabled on VD.

private const string BodyStateMapName = "VirtualDesktop.BodyState";
private const string BodyStateEventName = "VirtualDesktop.BodyStateEvent";

public unsafe struct BodyState
{
    #region Constants
    public const int ExpressionCount = 70;
    public const int ConfidenceCount = 2;
    public const int HandJointCount = 26;
    public const int FullBodyJointCount = 84;
    #endregion

    #region Fields
    // Face Tracking Data, XR_FB_face_tracking2
    public bool FaceIsValid;
    public bool IsEyeFollowingBlendshapesValid;
    public fixed float ExpressionWeights[ExpressionCount];
    public fixed float ExpressionConfidences[ConfidenceCount];

    // Eye Tracking Data, XR_FB_eye_tracking_social
    public bool LeftEyeIsValid;
    public bool RightEyeIsValid;
    public Pose LeftEyePose;
    public Pose RightEyePose;
    public float LeftEyeConfidence;
    public float RightEyeConfidence;

    // Hand Tracking Data, XR_EXT_hand_tracking
    public bool LeftHandActive;
    public bool RightHandActive;
    public fixed FingerJointState LeftHandJointStates[HandJointCount];
    public fixed FingerJointState RightHandJointStates[HandJointCount];

    // Hand Aim Tracking, XR_FB_hand_tracking_aim
    public HandTrackingAimState LeftAimState;
    public HandTrackingAimState RightAimState;

    // Body Tracking Data, XR_FB_body_tracking, XR_META_body_tracking_full_body, XR_META_body_tracking_fidelity
    public bool BodyTrackingCalibrated;
    public bool BodyTrackingHighFidelity;
    public float BodyTrackingConfidence;
    public fixed BodyJointLocation BodyJoints[FullBodyJointCount];
    public fixed SkeletonJoint SkeletonJoints[FullBodyJointCount];
    public int SkeletonChangedCount;
    #endregion
}

public struct FingerJointState
{
    #region Fields
    public Pose Pose;
    public float Radius;
    public Vector3 AngularVelocity;
    public Vector3 LinearVelocity;
    #endregion
}

public struct HandTrackingAimState
{
    public long AimStatus;
    public Pose AimPose;
    public float PinchStrengthIndex;
    public float PinchStrengthMiddle;
    public float PinchStrengthRing;
    public float PinchStrengthLittle;
}

public struct BodyJointLocation
{
    #region Fields
    public ulong LocationFlags;
    public Pose Pose;
    #endregion
}

public struct SkeletonJoint
{
    #region Fields
    public int Joint;
    public int ParentJoint;
    public Pose Pose;
    #endregion
}

public struct Pose
{
    #region Fields
    public Quaternion Orientation;
    public Vector3 Position;
    #endregion
}

public struct Quaternion
{
    #region Fields
    public float X;
    public float Y;
    public float Z;
    public float W;
    #endregion
}

public struct Vector3
{
    #region Fields
    public float X;
    public float Y;
    public float Z;
    #endregion
}

[shiftyscales]

I joined their Discord and found some additional information in a support thread- in addition to the above, there are several example implementations- hopefully this helps, @Frooxius.:

For references on how to read this data you can use the following: C++: https://github.com/mbucchia/VirtualDesktop-OpenXR/blob/0d2a45704aa847f62d819cdbc546237b7c235ccb/virtualdesktop-openxr/system.cpp#L561 C#: https://github.com/guygodin/VirtualDesktop.VRCFaceTracking/blob/main/TrackingModule.cs#L69 Python: https://pypi.org/project/PyVDLib

[shiftyscales]

The developer had also made the following comments/recommendations on this matter:

ggodin | Developer — Today at 15:42 I would recommend to use OpenXR rather than the non-public MMF ggodin | Developer — Today at 15:43 games can query all the face/eye/hand tracking through OpenXR extensions

[Frooxius]

Thanks for the info! I'll just work with the code samples then.

We can't use OpenXR at the moment unfortunately.

[DenTechs]

Another note from Guy is that the MMF can/will be changed in future updates which will break compatibility, which is why he recommended using OpenXR extensions if possible as those wont be changed.

[Frooxius]

I understand that, but like I mentioned, we can't use the OpenXR at the moment.

[DenTechs]

Yup, i'm just giving context for why he said that. :)