search

haixpham / end2end_AU_speech

Code for the paper "End-to-end Learning for 3D Facial Animation from Speech"
MIT License
70 stars 24 forks source link

Rendering a 3D model #2

Open JunZhang0629 opened 5 years ago

JunZhang0629 commented 5 years ago

Hi haixpham, I have some trouble when reproduce your code. How should I use a trained model to drive the face model? There is no details in your paper. And can I drive in real time? Waiting for your reply, thanks a lot .

haixpham commented 5 years ago

You must use the generic face model I provided in "shapes" folder. The details of shape rendering are in ShapeUtil2.py. If you want to use other languages, you have to use numpy to load and save it to a more suitable format, e.g. csv.

My code here takes on a whole sequence for processing, obviously to avoid the memory allocation overhead. But, you can run frame-by-frame in real time, by feeding the model one frame at a time. However, I do not recommend doing that in Python. The beauty of CNTK is that it is written in C++, so you can call CNTK model in your C++ application and enjoy real real-time performance.

JunZhang0629 commented 5 years ago

Hi, Hai Xuan Pham. Thanks for your reply very much.

I think your work is very interesting, so I still have some problems about end2end_AU_speech model. Firstly, how do you get the label data when you train the model? If there is no manual process of labeling data, will there be two stages of error accumulation: data preprocessing (or labeling) and training process. Secondly, in your code, what does the input video sequence do? Is the video sequence just used to extract the label? Waiting for your reply, thanks a lot .

------------------ 原始邮件 ------------------ 发件人: "Hai Xuan Pham"notifications@github.com; 发送时间: 2019年7月11日(星期四) 下午5:48 收件人: "haixpham/end2end_AU_speech"end2end_AU_speech@noreply.github.com; 抄送: "adolph"623224624@qq.com;"Author"author@noreply.github.com; 主题: Re: [haixpham/end2end_AU_speech] Rendering a 3D model (#2)

You must use the generic face model I provided in "shapes" folder. If you want to use in other languages, you have to use numpy to load and save it to a more suitable formats, e.g. csv.

My code here takes on a whole sequence for processing, obviously to avoid the memory allocation overhead. But, you can run frame-by-frame in real time, by feeding the model one frame at a time. However, I do not recommend doing that in Python. The beauty of CNTK is that it is written in C++, so you can call CNTK model in your C++ application and enjoy real real-time performance.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

haixpham commented 5 years ago

  1. I use my private tracker to track AU coefficients. You can find the reason why I can't give you my tracker here: #3. Details of my tracker can be found in my other papers.

  2. Yes, there are different sources of error: labeling and model training. Labeling is automatic, I believe it is standard practice in most datasets. I have to trust my tracker doing a good job.

  3. The video sequence is used to render the ground truth face along side 3D face shape. You can leave it as "None".