search

sign / translate

Effortless Real-Time Sign Language Translation
https://sign.mt
Other
414 stars 74 forks source link

Human Model is not working #124

Closed imsamimalik closed 6 months ago

imsamimalik commented 7 months ago

Current Behavior

image

When I select the human option from the dropdown, it just keeps on loading.

AmitMY commented 7 months ago

This is what I am seeing: image

What browser are you using? Are there any errors in the console? (https://documentation.concretecms.org/tutorials/how-open-browser-console-view-errors)

imsamimalik commented 6 months ago

My bad. There was some issue with Chrome Linux. It is working fine on Windows.

imsamimalik commented 6 months ago

For The 3D model, have you looked into implementing something like Ready Player Me API? Since you are already using Mediapipe, it would be fairly easy to implement it using MediaPipe Holistic landmarks.

I am currently working on making a 3D character animated from a source video using MediaPipe. Let me know if I can be of any help.

AmitMY commented 6 months ago

I haven't looked at their API before, but I also don't see anything implying they have an API to rig using MediaPipe? Could you point me in the right direction? I'd love to include it of course if possible.

imsamimalik commented 6 months ago

I got intrigued when I found this demo. He is extracting FaceLandmarks using MediaPipe and then using those landmarks he is Ready Player Me GLB model in the threejs.

I thought that if FaceLandmaks are possible then PoseLandmarks can be animated as well.

I also had a chance to look at Kalidokit. He is using VRM 3D models instead of Ready Player Me but the approach is the same (extracting full body landmarks using media pipe and then animating characters.)

For the 3D animation then you wouldn't need to send the whole file. At the Python backend, you can just extract the landmarks from the video and send those landmarks array as API response.

AmitMY commented 6 months ago

For the face demo, they are not using landmarks. Instead, they are using blendshapes.

Kalidokit is more promising to work directly from landmarks.

imsamimalik commented 6 months ago

Sorry, forgot about the blendshapes. But, when the new Holistic API gets released(which should be in a few weeks, according to this), Kalidokit + MediaPipe Holistic + Ready Player Me seems like the meta for me.

AmitMY commented 6 months ago

Closing since we already have an issue about Kalidokit #121