Full body XR avatars

[no-1ne]

👋 Sarge with the release of https://docs.readyplayer.me/ready-player-me/api-reference/avatars/full-body-xr-avatars looking forward to full body tracking using mediapipe holistic

It is currently in alpha with gpu support support coming soon

At the moment, there seems to be no option to obtain the facial transformation matrix which we can overcome by getting the rotation of face from facial landmarks with little math https://github.com/yeemachine/kalidokit/blob/main/src/FaceSolver/calcHead.ts

const filesetResolver = await FilesetResolver.forVisionTasks("https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision/wasm");

const holisticoptions = {
      runningMode: "VIDEO",
      outputFaceBlendshapes: true,
      outputPoseSegmentationMasks: false
    };
     holisticLandmarker = await HolisticLandmarker.createFromModelPath(filesetResolver,
      "https://storage.googleapis.com/mediapipe-models/holistic_landmarker/holistic_landmarker/float16/latest/holistic_landmarker.task"
  );
  holisticLandmarker.setOptions(holisticoptions);

const landmarks = holisticLandmarker.detectVideo(video);