search

TianxingWu / OpenVHead

A 3D virtual head control system for VTuber in Unity with smooth motion and robust facial expressions
MIT License
442 stars 55 forks source link

Facial tracking is not very good. #6

Closed delebash closed 4 years ago

delebash commented 4 years ago

First nice job on the project. I really like the ability to use new models via Blend Shapes. If you haven't looked at OpenFace. It is very responsive but the output is based on FACS AU. Another project FACSvatar which uses OpenFace connects FACS AU data to a Blender model but I don't see an easy way to use my own mode.

So I think your project is really what I am looking for but the facial landmark recognition is not really good enough to use.

Thanks!!!

TianxingWu commented 4 years ago

Thanks for your interest in this project! Yes, the face landmarks detection is quite noisy, since I just use the demo model file from the Dlib example programs, namely 'shape_predictor_68_face_landmarks.dat' , without customization.

To address this issue, I tried to add some filters and brought some simple ideas from control theory to reduce the noise, and construct some hand-crafted functions to make the facial expressions looks more realistic (see Readme 4.1.1 and 4.2.2).

However, it's still a compromise as the detection accuracy cannot be ''actually'' improved by these post-processing methods, and the optimal value of filter parameters highly depend on the device you use. So if you really want to use this project in practice with better performance, I would suggest two possible solutions: Firstly, try to tune the parameters of the filters and the PD control system; Secondly, replace the '.dat' file here with some better-trained models that can adapt with Dlib (and then tune the filter parameters again).

Good luck! :)

delebash commented 4 years ago

Thanks!