Make Avatars more realistic

[Mortezanavidi]

Hi, first of all i want to thank you for open sourcing this great project, i have already read other issues regarding the obstacles in the way to make the models more realistic ( using the same mesh and mixamo compatible bone structure as readyplayerme and having the arkit and oculus.

But i have tried to my best to create a better model from scratch but i was unsuccessful, either the structure is mixamo compatible but not compatible with the library ( i belive it has something to do with the geometry of the model, the second it get's imported and the idle animations kick in, i can see the idle animations are not normal at all ).

And the lips would not move at all even a bit when the lip sync gets triggered, now i wanted to ask, is there any kind of example model which is outside of websites like readyplayerme or avaturn that can be reproducible? ( we could use their mesh and the modules and just change the bone structure and texture )

[met4citizen]

Hi. Unfortunately, I don't have any realistic model that could be used as a template. As of now, I have only used the class with Ready Player Me, Avaturn, and Mixamo characters.

The idle poses used in the TalkingHead class are based on Mixamo poses, so you could try using Mixamo for rigging, and then see how your custom model works with Mixamo poses/animations on their site. If that works, you can download and rename the bones (basically just by removing the "mixamorig:" prefix).

Note that the available Mixamo characters don't come with the needed blend shapes (ARKit/Oculus), and without them facial expressions and lip-sync won't work. Additionally, most of their characters were not designed to support moving eyes and lips. Therefore, I suggest using RPM models as a reference/template.

I'm no expert in 3D modeling, but if you think you have everything in place and it still doesn't work, maybe you can give a link to your GLB file so that I (or someone else reading this) can take a closer look. Without the model, it's hard to say what the problem might be.

[adamiprinciples]

So I'm working on getting this working with characters created with Realusions Character Creator 4. It's not been easy but I'm pretty close now I think. The tricky parts have been mapping the meta visemes to CC4 standard and adding support for jaw bone rotation to some of them. The result is pretty decent and should unlock the way for more realistic character options. (Character can look much better than in this screenshot with better lighting etc)

[met4citizen]

This looks amazing! I haven't tried Reallusion CC myself, but now I guess I have to take a closer look. Well done, Adam! I can't wait to see and hear more. I have received many requests for more realistic avatars, so I'm sure I'm not the only one interested.

[adamiprinciples]

What I am struggling with is removing all of your pose stuff without breaking everything. Ideally I just want to use the pose that comes with the model as my plan is to just loop an external idle animation over the top. I've renamed all the bones but obviously the ready player me skeleton is quite different so things look a little odd

[met4citizen]

Yes, these CC models/rigs and their workflows are very different from RPM+Mixamo, so it might not be worthwhile to try to force them into the same code. Nevertheless, understanding what works and what doesn't is valuable information in itself, so I hope you keep us informed of your progress or if you decide to set up your own repo.

Regarding poses, one approach could be to auto-rig your CC model in Mixamo, although I have no idea how well that would turn out. On the other hand, if you already have an idle animation for the CC rig, then you're probably better off sticking with it in your use case.