search

pkhungurn / talking-head-anime-2-demo

Demo programs for the Talking Head Anime from a Single Image 2: More Expressive project.
http://pkhungurn.github.io/talking-head-anime-2/
MIT License
1.12k stars 154 forks source link

Choice of using iOS app iFacialMocap? #19

Closed bycloudai closed 2 years ago

bycloudai commented 2 years ago

Hi! Thank you for the cool work that you've published.

Is there a particular reason that you decided to use iFacialMocap? I'm just curious that is it because you think monocular cameras like webcams aren't as accurate as iPhone's depth cam?

dragonmeteor commented 2 years ago

The neural network in this project has many parameters that are used to control facial expressions. In order to be able to control a character's facial expression with your facial movement, you need a way to derive these parameters from some kind of video capture.

I tried to solve the problem of deriving these parameters from a normal monocular video capture, but I failed. See #2.

On the other hand, a standard technology stack for controlling characters with facial movement is to use an iPhone that comes with ARKit to capture your facial movement. ARKit can output blendShapes (https://developer.apple.com/documentation/arkit/arfaceanchor/2928251-blendshapes), which are a collection of 50 parameters that be used to control facial expression of 3D models. I found that it was very easy to derive my model's parameters from the blendShapes. The key technology is ARKit, which belongs to Apple. It is unfortunate that Android's ARCore does not have this feature. I also do not know any APIs that are available for free that can be used to infer the blendshape parameters from a monocular video capture.

iFacialMocap is just a software that streams blendShapes parameters from ARKit to my PC. There are other applications that can do the same job. I just happened to find it first from a Google search.