Documentation

[ShakeelKhalid1913]

No documentation? I want to understand the code and its been a month. and because of no documentation, I didn't understand anything.

[AmitMY]

Do you have any specific questions? I could add documentation, I just need to know what is not clear

[ShakeelKhalid1913]

I want to know how character animation is done. everything related to it. Three js is used but how does it work? also which model is used to fetch landmarks points (holistic or something else)?

in simple words, i am working on this kind of project and i want to know all details. how can i animate a character in three js with points.

if I'm not wrong, is these are steps how this project works?

1. Download dataset of videos of signlanguage
2. of every word from dataset get its points and store in database
3. then if some one request a sentence. it generates its glossary and get all those points that exist in database
4. then using machine learning model, connect those points. so it look realistic. and animate on character
5. and show to user

I am stuck on two steps.

• first is which model should i use to fetch points (holistic or some other).
• secondly how to animate characters through three js (I'm using mixamo character)

i need answers of these two questions. i have been researching on more than 7 months and still not got any solution. Can you help me also with explanation of code by adding comments or documentation on which functions could help me to understand it better.

thank you :)

[AmitMY]

Seems like what you care about specifically is the pose estimation and mixamo animation system (4). I have made a video about it: https://youtu.be/TyJuU9_GOaw As you can see from the video, the character does not follow the pose correctly. You can try it yourself here: https://sign.mt/playground (Hit "Turn on Camera" and activate "Animate pose")

Holistic is used to get the keypoints, and the animation with Mixamo is just a model I trained.

To move a mixamo model in the web, the process is as done here: https://github.com/sign/translate/blob/master/src/app/components/animation/animation.component.ts#L58-L78

---

These days I am writing a paper that improves on the model in the video. When I finish, and when it is stable, I hope to have a better model that actually animates correctly based on the pose

[ShakeelKhalid1913]

Great. I really appreciate your help.

We were trying to animate the character, but it did not work correctly because of Animating Quaternion Rotation. Can you provide me with the model you used to animate the Mixamo in this repo? or any tutorial online by which I can do animation from pose estimation points. I need a tutorial on sign animation of some English essential words like Hello, Good, and You with code explanation including Animating Quaternion Rotation in three js whether it could be a simple node js project or React (Also the main issue is I'm not angular guy which is big problem xD).

We are students and this is our final year BS computer science project. We have researched many papers. we have checked HandTalk mobile application too. but they didn't even provide any explanation of how they did. It would be great if you can help to share tutorial with us.

[AmitMY]

I won't be able to make a tutorial, the YouTube video is the best I can do for you.

Pose service: https://github.com/sign/translate/blob/master/src/app/modules/pose/pose.service.ts Animation service: https://github.com/sign/translate/blob/master/src/app/modules/animation/animation.service.ts Model is in src/assets/models/pose-animation/model.json

[AmitMY]

For other models to animate avatars from mediapipe: https://github.com/yeemachine/kalidokit

[ShakeelKhalid1913]

Great. Thank you for your help :)