search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.58k stars 5.16k forks source link

Feature Request: Full demo application that can be used for 3D character animation. #1293

Closed delebash closed 3 years ago

delebash commented 4 years ago

By combining your existing 3D body motion capture applications into one application that can stream real-time data for use in a 3D character animation program such as Maya or Blender. I think this would be a wonderful showcase application of what mediapipe can do.

sgowroji commented 3 years ago

Hi @delebash, Thank you sharing your interest on MediaPipe. Could you please elaborate your use case with more details and who will benefit with this future. Are you willing to contribute it. Thanks!

delebash commented 3 years ago

Sure, anyone wanting to do 3D motion capture in say Unreal for video games or movies would find it extremely useful. Not just Unreal but any 3D animation software, Maya, Blender, iClone. Here is a test I did with iClone but because of mediapipe still having jitter problems it is just not feasible yet. https://www.youtube.com/watch?v=j6JboJIlpfM. There are commercial companies doing this now and it would be nice to have an open source solution for those of us who cannot afford it. See https://getrad.co/ or https://www.deepmotion.com/. Nvidia is using Wrnch AI for their motion capture application that is part of a 3D suite of apps https://developer.nvidia.com/blog/nvidia-omniverse-machinima-now-available-in-open-beta/. Unreal has a free iPhone app for facial motion capture but you must have a depth sensing camera Iphone 11> and only face is tracked. Being able to do full body motion capture from a web camera would help solo developers and small companies who cannot afford the expensive motion capture body suits and other equipment be able to compete in creating AAA games and CGI Movies. As far as contributing I am not a really good coder but I can test the product and give feedback that is certainly not a problem. Currently one of the biggest problems with mediapipes computer vision apps is the quality. Take a look at this post https://github.com/google/mediapipe/issues/825 the jitter happens in the hands face, and body detection as well. If you watch my youtube post you will notice that when I am not moving my fingers mediapipe still errors by detecting motion. The is easily seen in any of your hands, pose, face web demos, same happens on python so it is not a javascript problem. But as you can see other companies have solved this problem somehow. Maybe better datasets or filter algorithms , this is where I lack the programming knowledge to help solve this problem.

francoisbonnard commented 3 years ago

Hi @sgowroji . Following the idea of Daniel @delebash I would like to help if needed for Maya. My concern is more about facial meshing and how to import data inside Maya so we can adjust 3D models on an image sequence (or real time)

sgowroji commented 3 years ago

Hi @francoisbonnard, This https://github.com/google/mediapipe/issues/1155#issuecomment-721889436 will guide you better on your use case. Thanks!