search

haruiz / RiggingJs

Tensorflow.js and Three.js skeletal animation tool
Apache License 2.0
271 stars 32 forks source link

incorporating POSENET #2

Open kavikode opened 4 years ago

kavikode commented 4 years ago

You have an excellent demo! This is what I have been looking for many years!

Can you please include the following to move the arms and limbs of the model in 3D with 2 web cameras?

https://youtu.be/7lMOdYc4F4c

Thank you!

haruiz commented 4 years ago

Hi @kavikode, I 'm currently working on that implementation, so I hope to have this done by next week, thanks for sharing that amazing demo, now is already in the roadmap: #2

kavikode commented 4 years ago

Thank you so much @haruiz ! I can't wait!

Deamoner commented 4 years ago

Yo, how is this looking? Do you have any architecture notes? I could jump in and help with this.

haruiz commented 4 years ago

Hey @Deamoner, I appreciate your interest, the idea is to use the posenet tensorflow.js model to animate the model, you can get more information on the roadmap of the project , I'm currently working on it, but you are welcome to contribute with that, let me know if you need anything, we can chat or discuss about it.

kavikode commented 4 years ago

@Deamoner are you interested in helping with a certain aspect? Can you please help to create the IK for each of limbs so that when you use one of the keypoints from PoseNet then the rest of the joints follow the selected IK of the limb?

Please see the following demo: https://rawgit.com/takahirox/three.js/GLTFMMD/examples/webgl_loader_gltf_miku_ik.html

Deamoner commented 4 years ago

Let me review. Do you have a discord or chat?

Sent from my iPhone

On Jun 25, 2020, at 7:20 AM, kavikode notifications@github.com wrote:

 @Deamoner are you interested in helping with a certain aspect? Can you please help to create the IK for each of limbs so that when you use one of the keypoints from PoseNet then the rest of the joints follow the selected IK of the limb?

Please see the following demo: https://rawgit.com/takahirox/three.js/GLTFMMD/examples/webgl_loader_gltf_miku_ik.html

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

kavikode commented 4 years ago

@Deamoner thanks! I'm on discord Kavita Krishnaswamy#5836. my email kavi1@umbc.edu

haruiz commented 4 years ago

@Deamoner and @kavikode, I could create a Slack channel, so we can talk more in detail.

kavikode commented 4 years ago

@haruiz and @Deamoner that sounds like a wonderful plan! Please do email me the details. I feel so excited because I have been independently trying to solve this problem for more than 2 years. Looking forward to our discussion.

Deamoner commented 4 years ago

Feel free to just put the link here and I'll join. I'm just finishing up a posenet dance app.

haruiz commented 4 years ago

@Deamoner , @kavikode guys, this is the link to access to the slack channel: https://join.slack.com/t/riggingjs/shared_invite/zt-fcjr9p12-GqIj7zEjoNnH7j5AwWT7MQ

kavikode commented 3 years ago

@haruiz I'd like to port 3D Human Pose and Mesh Estimation to Tensorflow.js.

Please see the following: https://youtu.be/tDz2wTixcrI

Can you please help me port it to Tensorflow? This may solve some of the problems.

haruiz commented 3 years ago

@haruiz I'd like to port 3D Human Pose and Mesh Estimation to Tensorflow.js.

Please see the following: https://youtu.be/tDz2wTixcrI

Can you please help me port it to Tensorflow? This may solve some of the problems.

@kavikode let's do it, I will create a new repo for it, so we can start working on it!

delebash commented 3 years ago

Hi all @haruiz I modified your version and was able to stream the data for hand pose to my character in a 3D program and it works. The problem is the hand detection and joint accuracy. It is just not good enough for real use in animation of a 3D character. Take a look at this project https://github.com/vladmandic/human. He has made significant improvements on the default tfjs version. I have also discussed with him about making another version that would use better datasets and bounding box detection for greater stability and accuracy. My current project concept is to be able to stream the raw data where any application can pick it up via its own plugin and format the data for its animation system. This leaves the whole motion capture loosely coupled from any other software. But thank you for your software as I have used it to test if this can be done. Well done!!!!

haruiz commented 3 years ago

Thanks so much, @delebash for your feedback, I love that you found rigging.js useful, in the next few weeks I will put all my effort to make good progress, and try to implement the rest of the features that are in the roadmap, so I will follow up with your recommendations and I'll consider them for improving rigging.js and release a more complete version!!

ubilrodriguez commented 3 years ago

Excuse me, you can send me that avatar, I am doing an investment job, because the project does not find that avatar, there is only a sample avatar.