Ready Player Me Half-Body Avatars

[z3rec]

Description We have some old Ready Player Me Avatars (from October 2022) that are working perfectly fine. But since January the Avatars are Bugging in VR. The Animations, as well as the Avatar Meshes are not working correctly. (Lower half of the Screenshot) The Mesh has same weird holes and the Animations are flicking back to the T-Pose while animating. It looks like some Frames in the Animations are set to T-Pose. I opened them both in Blender, but they are the same. I cannot see any difference that causes such a big Issue. At the Moment the Ready Player Me Avatars are not usable for our hubs-cloud instance [as-well as for the offical hubs.mozilla.com]

To Reproduce Steps to reproduce the behavior:

1. Go to 'https://baermedia.readyplayer.me/de/avatar'
2. Create yourself some sort of Avatar (without Glasses, because they are bugged too)
3. Got to hubs.mozilla.com
4. Log yourself in
5. Change your Avatar to your created one from step 2
6. Enter in VR

Expected behavior (Upper Half of the Screenshot) The Avatar has the standard VR Hands that are Animating correct and synchronized with the Users Hands

Screenshots

Hardware

• Device: VR headset
• OS: Windows
• Browser: Chrome

--Edit Additional Information I just saw that all working Avatars have only Quaternion Tracks on it.

All Broken Avatars have many others with Vectors.

I dont know if it correlates to the reason why its broken, but its the first Difference i figuered out so far.

[MrOrz]

Some details of the observed behavior of the glitch:

• The user that wears the glitched avatar will:
  • see their own hands in broken mesh
  • hands would flicker when triggering the finger animations
• Other user that looks at the user with the glitched avatar will:
  • see the glitched user's hand intact (mesh not broken in this case)
  • hands flicking back to T-pose when finger animation is playing

[matthewbcool]

@MrOrz Thank you so much for this detailed report. As you already helped everyone figure out, there was that issue with generating multiple morph-audio-feedback on the RPM side that breaks the avatar in Hubs but I wonder if there is something we can do or should do about the hands on the Hubs side?

[z3rec]

At the moment the only Fix we found is importing the Skin into Blender and Exporting it again. It gets a few MB bigger, but we do not really have a clue why. It fixes the broken Meshs.

For the Animations we did a quick "Fix" and just deactivated all Hand Animations for the moment. If someone figures out a better Solution, please comment it to this Issue. The Hands without Animations are a bit boring, but way better than the flickering ones.

[MrOrz]

Just seen this post regarding frustum culling. Not sure if this can help the flicker. https://hubs.mozilla.com/labs/frustrations-with-frustum-calling-a-guide/

[NigelRidderhof]

Hi, we are also running into this issue, as you can see months later.

We used to mainly use older avatars that we created quite some time ago. These (still) do not have these weird flashing / glitching hands with holes in VR, but any new one that we make does. This is a problem especially because we built in the possibility to create your own RPM avatar within our platform using the RPM API. I'm not certain on what side the problem and therefore possible solution lies, RPM or Hubs. Note that it does not matter if you create an avatar through the API or the more general editor (https://vr.readyplayer.me/nl/avatar).

[CLB-ZReality]

@z3rec @MrOrz Thank you for this detailed report of the problem. I bumped into this issue too and your observations helped me a lot to fix this issue for us.

I wanted to share our fix, if anyone still has this issue and wants to resolve this. This only fixes the T-Pose flickering of the hands. The holes still persist, but can be fixed when importing and reexporting the Avatar glb from Blender as @z3rec pointed out.

So the fix for the flickering follows:

1. export the NETWORK_POSES from hand-poses.js (only needed to check if the animation is a hand pose) export const NETWORK_POSES = ["allOpen", "thumbDown", "indexDown", "mrpDown", "thumbsUp", "point", "allGrip", "pinch"];

2. filter out all unnecessary VertexKeyframeTracks (we only need the QuaternionKeyframeTracks for rotation). Afterwards filter the last 3 QuaternionKeyframeTracks of each tracks array too, to delete the rotation keyframes of the hand bone. This has to be done in animation-mixer.js initMixer() function.

   const leftRightFilter = new RegExp("(_L)|(_R)");
   //Fix hand pose animations by removing unnecessary VectorKeyframeTracks (Position and Scale)
   this.animations.forEach(animClip => {
     if (animClip && NETWORK_POSES.includes(animClip.name.replace(leftRightFilter, ""))) { //Check if this is really a hand pose
       //Filter out EVERY track that is NOT a quaternion track and therefor not rotation related
       animClip.tracks = animClip.tracks.filter(track => track.ValueTypeName == "quaternion");
       //Filter out the quaternion track of the Hand bone!!! Otherwise the hand still rotation flickers in place
       animClip.tracks.splice(animClip.tracks.length - 3, 3);
     }
   });