Strange character deformation for models imported from Vroid

[michaeljyang]

Hi there,

First of all, really appreciate you for this project. I got to mocap working with the examples. However, when I imported a model made with Vroid (https://vroid.com/en/studio), the character became very strange.

This is the great result I got with the “SK_Mannequin_Mobile” skeleton mesh:

So after I got good results with the “SK_Mannequin_Mobile” skeleton mesh, I changed the skeleton mesh in my character blueprint to a ue4mannequin type that showed up when I imported my vroid model with a plugin called VRM4U (https://github.com/ruyo/VRM4U/releases).

This is how my model should have looked like:

Then I re-assigning the skeleton to "SK_Mannequin_Mobile_Skeleton" in the MMVR file which lead to the following results.

This is how it turned out after re-assigning the skeleton to "SK_Mannequin_Mobile_Skeleton" in the MMVR file:

I'm not sure what is wrong with the skeleton. Would you please have a look at this and give me some advice? Thank you very much.

I'm using Unreal 5.03, vive trackers. The model file I imported from vroid is this one: Anya Forger.zip

[michaeljyang]

I also tried to follow the chapter about the strange deformation for metahumans. However, I'm not sure if i did it wrong but it didn't work for me.

[Megasteakman]

Do your bone axis alignments also match the UE4 mannequin? I'm using some bone orientations in the set up and IK solver phases, so it they aren't the same there could be really unpredictable results. If the orientations are different, I think this would be a good use-case for the IK retargeting actor profile: https://www.youtube.com/watch?v=ZBv5qi4GJDk&t=734s

[michaeljyang]

I'm really new to skeleton and Ureal so I'm not that sure. But when i reassigned the skeleton to the mesh, there were more joints in the mesh I imported than the skeleton in the MMVR file. I'll try to go through your IK retargeting first. Thanks!

[michaeljyang]

I actually have another question about finger tracking. I could only manage to get three fingers moving on each hand. My thumb and index finger weren't able to be tracked. Any guess what may have caused this?

[Megasteakman]

Yeah, the fact that the joints were mismatched is probably another problem.

Not sure why your fingers aren't working. The index is a little imprecise, and I find somehow I have to drum my fingers back and forth to get it working.

[michaeljyang]

OMG! IK retargeting works! I got my model to move! THANK YOU SO MUCH!

I have another question about the facial expressions. Does VRM imported models support facial expression? Or could I make some presets and get them to work in unreal?

[Megasteakman]

Hmm, I actually don't know how that works for other people: I think they use a lot of external apps to drive those? I did include a retargeting asset template in the project so you can make the AR Kit stuff remap to whatever blendshapes you want, but you may have to hard code the live link phone name into your characters anim bp and use that retargeting asset in there.

[michaeljyang]

Ok thanks man. That sounds a little bit to difficult for me now. By the way, I had some problems with the inputs of my index controllers. During calibration, I could rotate the target mesh but when I wanted to move it with my left controller it could only go oneway. Did this occur to you? Also, the controller inputs would just not work for sometimes. I need to restart the project/steam vr to get them working again.

[Megasteakman]

For inputs, you can try running the project in vr mode before you try running it in normal game mode, or flipping some of the input setting back and forth in project settings (change the setting from the correct button mapping to an incorrect one and back). Also, turning on the controllers before the trackers can help (sometimes the trackers assume the roles of controllers and steal their inputs).

I'm sure there's some kind of VRM face tutorial on youtube: I just haven't implemented them because I don't typically use those models and don't want to rely on anything outside of unreal engine for the project to work.

[michaeljyang]

Really appreciate your time and patience. For the trackers stealing controller part, I figured out that adding trackers first in live link then check the controllers would help : )