RPM-Face-Tracing in Godot using C#

[GeorgeS2019]

Import RPM Avatar to Godot

[GeorgeS2019]

Code Link

import { Canvas, useFrame, useGraph } from '@react-three/fiber';

 const { scene } = useGLTF(`${url}?morphTargets=ARKit&textureAtlas=1024`);
  const { nodes } = useGraph(scene);

  useEffect(() => {
    headMesh = (nodes.Wolf3D_Head || nodes.Wolf3D_Avatar);
  }, [nodes, url]);

Get BlendShape values and Head Rotation values

Wolf3D_Head (MeshInstance3D)

  useFrame((_, delta) => {
    if (headMesh?.morphTargetInfluences && blendshapes.length > 0) {

      blendshapes.forEach(element => {
        let index = headMesh.morphTargetDictionary[element.categoryName];
        if (index >= 0) {
          headMesh.morphTargetInfluences[index] = element.score;
        }
      });

      nodes.Head.rotation.set(rotation.x, rotation.y, rotation.z);
      nodes.Neck.rotation.set(rotation.x / 5 + 0.3, rotation.y / 5, rotation.z / 5);
      nodes.Spine2.rotation.set(rotation.x / 10, rotation.y / 10, rotation.z / 10);
    }
  });

https://github.com/srcnalt/rpm-face-tracking/assets/49812372/7203479e-9d34-4923-b4da-da10d068b5fd

Wolf3D_Head (MeshInstance3D)

Head, Neck, Spine2 rotations (Skeleton3D)

[srcnalt]

This is neither Godot, nor a C# project, I will have to close the issue. Feel free to use the code example to convert the logic into Godot/C#.

[GeorgeS2019]

@srcnalt It is fine to close it. We now have Mediapipe webcam capture to work, my plan is to engage more Godot users to automate Unity to Godot involving RPM.

[GeorgeS2019]

@srcnalt We now have working Godot4 Mediapiple with complete facial landmarks Godot GDMP

It would be great to have a demo using Godot GDMP to drive ReadyPlayerMe using RPM-Godot-SDK

[srcnalt]

Great work, I will check it when I get to my PC.