search

met4citizen / TalkingHead

Talking Head (3D): A JavaScript class for real-time lip-sync using Ready Player Me full-body 3D avatars.
MIT License
350 stars 108 forks source link

bad looking gesture when IK superposed on the base gesture #41

Closed branaway closed 5 months ago

branaway commented 6 months ago

When the base pose is "hip", the IK resolved to gestures with the influence from the base and they don't look good. See the left hand in the following snapshot:

Screen Shot 2024-05-14 at 13 47 35

branaway commented 6 months ago

Ideally the IK result should based on hand gestures in neutral positions.

met4citizen commented 6 months ago

I classified this issue as an enhancement (new feature) because methods touchAt and ikSolve are still more or less experimental. They are used in the test app, and you are free to use them, but I haven't included them in the README list of methods, and they have their shortcomings.

As for the hands, maybe the best approach going forward is to have a set of different hand poses/gestures to choose from. Neutral/relaxed hand, fist, pointing finger, thumbs up, etc. So, similar to poseTemplates (weight on left leg, right automatically generated), there could be handTemplates (left hand/finger rotations, right generated). And then, in some way, the app could set the left/right hand poses separately, and they could perhaps be included in the animations too.

I need to think about this some more, and if I proceed with different hand poses, implementing it will take some time.

branaway commented 6 months ago

Hi many thanks again. I was seeking some advice as how to go about it. My quick and dirty is to copy a natural arm quaternion to the arms before it goes on in the ikSolve. A value works for me is new Quaternion(.56, .24, .12, .68) .

met4citizen commented 6 months ago

OK, so you were talking about the position of the arm, not the hand/fingers as I originally thought. Yes, the inverse kinematics algorithm starts finding the direct route from the current position, so your solution probably works fine. Of course, what is considered natural depends on the situation, and perhaps preference. For me, for example, your screenshot looked like a perfectly natural pose. If you are trying to touch the same side of your body with your hand, you have to raise your arm. But I understand your point. Cheers.

branaway commented 6 months ago

Yes, the arm position came from the Hip pose. And it looked awkward. The finger gesture is something I'll work on soon. An example would be to show "OK" sign to the viewer.

met4citizen commented 5 months ago

I just added support for gestures, so I will close this issue. The default gestures include, for example, head.playGesture("ok");. Thank you for giving me the idea.

branaway commented 5 months ago

wow, will check it out! Thanks!