Make Avatars more realistic

[Mortezanavidi]

Hi, first of all i want to thank you for open sourcing this great project, i have already read other issues regarding the obstacles in the way to make the models more realistic ( using the same mesh and mixamo compatible bone structure as readyplayerme and having the arkit and oculus.

But i have tried to my best to create a better model from scratch but i was unsuccessful, either the structure is mixamo compatible but not compatible with the library ( i belive it has something to do with the geometry of the model, the second it get's imported and the idle animations kick in, i can see the idle animations are not normal at all ).

And the lips would not move at all even a bit when the lip sync gets triggered, now i wanted to ask, is there any kind of example model which is outside of websites like readyplayerme or avaturn that can be reproducible? ( we could use their mesh and the modules and just change the bone structure and texture )

[met4citizen]

Hi. Unfortunately, I don't have any realistic model that could be used as a template. As of now, I have only used the class with Ready Player Me, Avaturn, and Mixamo characters.

The idle poses used in the TalkingHead class are based on Mixamo poses, so you could try using Mixamo for rigging, and then see how your custom model works with Mixamo poses/animations on their site. If that works, you can download and rename the bones (basically just by removing the "mixamorig:" prefix).

Note that the available Mixamo characters don't come with the needed blend shapes (ARKit/Oculus), and without them facial expressions and lip-sync won't work. Additionally, most of their characters were not designed to support moving eyes and lips. Therefore, I suggest using RPM models as a reference/template.

I'm no expert in 3D modeling, but if you think you have everything in place and it still doesn't work, maybe you can give a link to your GLB file so that I (or someone else reading this) can take a closer look. Without the model, it's hard to say what the problem might be.

[adamiprinciples]

So I'm working on getting this working with characters created with Realusions Character Creator 4. It's not been easy but I'm pretty close now I think. The tricky parts have been mapping the meta visemes to CC4 standard and adding support for jaw bone rotation to some of them. The result is pretty decent and should unlock the way for more realistic character options. (Character can look much better than in this screenshot with better lighting etc)

[met4citizen]

This looks amazing! I haven't tried Reallusion CC myself, but now I guess I have to take a closer look. Well done, Adam! I can't wait to see and hear more. I have received many requests for more realistic avatars, so I'm sure I'm not the only one interested.

[adamiprinciples]

What I am struggling with is removing all of your pose stuff without breaking everything. Ideally I just want to use the pose that comes with the model as my plan is to just loop an external idle animation over the top. I've renamed all the bones but obviously the ready player me skeleton is quite different so things look a little odd

[met4citizen]

Yes, these CC models/rigs and their workflows are very different from RPM+Mixamo, so it might not be worthwhile to try to force them into the same code. Nevertheless, understanding what works and what doesn't is valuable information in itself, so I hope you keep us informed of your progress or if you decide to set up your own repo.

Regarding poses, one approach could be to auto-rig your CC model in Mixamo, although I have no idea how well that would turn out. On the other hand, if you already have an idle animation for the CC rig, then you're probably better off sticking with it in your use case.

[namnm]

@adamiprinciples how are you doing so far with the realistic models? I would love to see your final result!

[AkshatRastogi-1nC0re]

@adamiprinciples I want to work on this with you. Is it possible we could connect regarding this ?

[RobLucchi]

Hello @adamiprinciples ,

I hope you're doing well, Really Impressive work btw.

I was wondering if you could share an update on your progress or any tips you might have for someone trying to achieve similar results. Your insights would be incredibly valuable, and I’m sure many others in the community would appreciate it too.

Thank you.

[met4citizen]

If you are still looking for more realistic avatars, you might also want to check out Microsoft Rocketbox. Their models are a bit old, but they are now available under the MIT License, which is nice. The rig is slightly different, but the models come with ARKit blendshapes and Oculus visemes. If you know the basics of 3D modeling, it’s not that much work to make them compatible with the TalkingHead class. I just tried converting one of their models, and here's the result as a proof-of-concept:

https://youtu.be/1otBhGEoGGg

Here are the main steps needed:

1. Download the model: Get the Microsoft Rocketbox "facial" version along with the Texture folder and its contents.
2. Remove the existing rig using Blender: Import the model into Blender, apply all transforms to all objects, apply the pose as the rest pose, remove the rig, and export to FBX.
3. Auto-rig in Mixamo: Upload the rig-free FBX to Mixamo, auto-rig it, and download as an FBX with a T-pose.
4. Add/rename/fix in Blender: Import the rigged FBX into Blender, apply all transforms to all objects, rename bones and shape keys, add LeftEye/RightEye bones, and set the following for all meshes: material > surface > metallic = 0. Fine-tune the model, then export it to GLB.

You can rename the bones and shape keys manually, but I wrote a short script for Blender (just within the limits of my Python skills):

import bpy, re

# Shape key map for Microsoft Rocketbox
shapekeyMap = [
[ "AA_VI_00_Sil", "viseme_sil" ],
[ "AA_VI_01_PP", "viseme_PP" ],
[ "AA_VI_02_FF", "viseme_FF" ],
[ "AA_VI_03_TH", "viseme_TH" ],
[ "AA_VI_04_DD", "viseme_DD" ],
[ "AA_VI_05_KK", "viseme_kk" ],
[ "AA_VI_06_CH", "viseme_CH" ],
[ "AA_VI_07_SS", "viseme_SS" ],
[ "AA_VI_08_nn", "viseme_nn" ],
[ "AA_VI_09_RR", "viseme_RR" ],
[ "AA_VI_10_aa", "viseme_aa" ],
[ "AA_VI_11_E", "viseme_E" ],
[ "AA_VI_12_I", "viseme_I" ],
[ "AA_VI_13_O", "viseme_O" ],
[ "AA_VI_14_U", "viseme_U" ],
[ "AK_01_BrowDownLeft", "browDownLeft" ],
[ "AK_02_BrowDownRight", "browDownRight" ],
[ "AK_03_BrowInnerUp", "browInnerUp" ],
[ "AK_04_BrowOuterUpLeft", "browOuterUpLeft" ],
[ "AK_05_BrowOuterUpRight", "browOuterUpRight" ],
[ "AK_06_CheekPuff", "cheekPuff" ],
[ "AK_07_CheekSquintLeft", "cheekSquintLeft" ],
[ "AK_08_CheekSquintRight", "cheekSquintRight" ],
[ "AK_09_EyeBlinkLeft", "eyeBlinkLeft" ],
[ "AK_10_EyeBlinkRight", "eyeBlinkRight" ],
[ "AK_11_EyeLookDownLeft", "eyeLookDownLeft" ],
[ "AK_12_EyeLookDownRight", "eyeLookDownRight" ],
[ "AK_13_EyeLookInLeft", "eyeLookInLeft" ],
[ "AK_14_EyeLookInRight", "eyeLookInRight" ],
[ "AK_15_EyeLookOutLeft", "eyeLookOutLeft" ],
[ "AK_16_EyeLookOutRight", "eyeLookOutRight" ],
[ "AK_17_EyeLookUpLeft", "eyeLookUpLeft" ],
[ "AK_18_EyeLookUpRight", "eyeLookUpRight" ],
[ "AK_19_EyeSquintLeft", "eyeSquintLeft" ],
[ "AK_20_EyeSquintRight", "eyeSquintRight" ],
[ "AK_21_EyeWideLeft", "eyeWideLeft" ],
[ "AK_22_EyeWideRight", "eyeWideRight" ],
[ "AK_23_JawForward", "jawForward" ],
[ "AK_24_JawLeft", "jawLeft" ],
[ "AK_25_JawOpen", "jawOpen" ],
[ "AK_26_JawRight", "jawRight" ],
[ "AK_27_MouthClose", "mouthClose" ],
[ "AK_28_MouthDimpleLeft", "mouthDimpleLeft" ],
[ "AK_29_MouthDimpleRight", "mouthDimpleRight" ],
[ "AK_30_MouthFrownLeft", "mouthFrownLeft" ],
[ "AK_31_MouthFrownRight", "mouthFrownRight" ],
[ "AK_32_MouthFunnel", "mouthFunnel" ],
[ "AK_33_MouthLeft", "mouthLeft" ],
[ "AK_34_MouthLowerDownLeft", "mouthLowerDownLeft" ],
[ "AK_35_MouthLowerDownRight", "mouthLowerDownRight" ],
[ "AK_36_MouthPressLeft", "mouthPressLeft" ],
[ "AK_37_MouthPressRight", "mouthPressRight" ],
[ "AK_38_MouthPucker", "mouthPucker" ],
[ "AK_39_MouthRight", "mouthRight" ],
[ "AK_40_MouthRollLower", "mouthRollLower" ],
[ "AK_41_MouthRollUpper", "mouthRollUpper" ],
[ "AK_42_MouthShrugLower", "mouthShrugLower" ],
[ "AK_43_MouthShrugUpper", "mouthShrugUpper" ],
[ "AK_44_MouthSmileLeft", "mouthSmileLeft" ],
[ "AK_45_MouthSmileRight", "mouthSmileRight" ],
[ "AK_46_MouthStretchLeft", "mouthStretchLeft" ],
[ "AK_47_MouthStretchRight", "mouthStretchRight" ],
[ "AK_48_MouthUpperUpLeft", "mouthUpperUpLeft" ],
[ "AK_49_MouthUpperUpRight", "mouthUpperUpRight" ],
[ "AK_50_NoseSneerLeft", "noseSneerLeft" ],
[ "AK_51_NoseSneerRight", "noseSneerRight" ],
[ "AK_52_TongueOut", "tongueOut" ],
[ "AU_26_JawDrop", "mouthOpen" ],
[ "HB_07_MouthSmile", "mouthSmile" ],
[ "AU_45_Blink", "eyesClosed" ],
[ "AU_63_EyesUp", "eyesLookUp" ],
[ "AU_64_EyesDown", "eyesLookDown" ]
]

# Recursive traverse
def traverse(x):
    yield x
    if hasattr(x, 'children'):
        for c in x.children:
            yield from traverse(c)

# Traverse objects
for r in bpy.context.scene.objects:
    for o in traverse(r):
        if hasattr(o, 'data'):
            if hasattr(o.data, 'bones'):

                # Rename mixamo bones
                for b in o.data.bones:
                    print(b.name);
                    b.name = re.sub(r'mixamorig:', '', b.name)

            if hasattr(o.data, 'shape_keys'):
                if hasattr(o.data.shape_keys, 'key_blocks'):

                    # Rename shape keys
                    keys = o.data.shape_keys.key_blocks
                    for m in shapekeyMap:
                        idx = keys.find(m[0])
                        if idx != -1:
                            print(idx)
                            keys[idx].name = m[1] 

In principle, you could use the same procedure for CC models. However, the Character Creator 4 software only works on Windows (I have a Mac), and at least the free model I downloaded didn’t include Oculus viseme support. In fact, Reallusion seems to have made every effort (both technically and legally) to lock you into their product ecosystem. If you plan on using their products or models, be sure to read their content end-user license agreement carefully.