search

zhangchenxu528 / FACIAL

FACIAL: Synthesizing Dynamic Talking Face With Implicit Attribute Learning. ICCV, 2021.
GNU Affero General Public License v3.0
377 stars 83 forks source link

Audio not in sync #37

Open Vidyaranya opened 2 years ago

Vidyaranya commented 2 years ago

I am giving a new audio file as input and trying to re-generate an Obama audio using Colab but it still seems to take the pose corresponding to the original audio file present in the Colab and the lips are completely out of sync.

I have regenerated a new pkl file from the new audio file using audio_preprocessing and then used test.py to get the .npz file. I have used the remaining models as they are present already in the colab demo. Is there something else that needs to be changed?

zhangchenxu528 commented 2 years ago

Are you using the audio file from Obama or someone else? We train the audio2face network only for Obama. If you use other people's audios as test examples, you need to train the specific audio2face network first.

Can you show some results here? So we find where the problem is.

Vidyaranya commented 2 years ago

Thanks for the response @zhangchenxu528 I am using Biden's voice with Obama's picture. I assumed that the other person's voice also works as the demo colab notebook indeed has a female voice at the start with Obama's picture. The results can be found here

zhangchenxu528 commented 2 years ago

The second half of the video is fixed, which means that you are using the original demo result when running rendering _gaosi. Eg: 'python rendering_gaosi.py --net_params_path ../examples/test-result/test_new.npz', please modify the net_params_path in section 4.2.

zhangchenxu528 commented 2 years ago

To verify your results, I used the audio you provided for testing. Test result: https://drive.google.com/file/d/1654tGYjxMDiRAjN-lR9CMlSQ9x094vQf/view?usp=sharing