bad lip sync

[kike-0304]

I used the model files and preprocessed data provided by the author, but the results were very unsatisfactory. The mouth shape was completely inaccurate, and there was a significant gap compared to Genface and Ernrf.

[KylinYee]

Please provide more results or details so that we can locate the problem more quickly.

[kike-0304]

Please provide more results or details so that we can locate the problem more quickly. run test_pretrained.sh restlt: https://drive.google.com/file/d/1vbTvDyof3IhETsmkTfh1rCIyedyLzIaU/view?usp=sharing

[kike-0304]

Please provide more results or details so that we can locate the problem more quickly. run test_pretrained.sh restlt: https://drive.google.com/file/d/1vbTvDyof3IhETsmkTfh1rCIyedyLzIaU/view?usp=sharing method=r2talker # r2talker, genefaceDagger, rad-nerf cond_type=idexp # eo, idexp vid=Obama

python test.py \ --method ${method} \ --cond_type ${cond_type} \ --pose ./pretrained/transformsval.json \ --ckpt ./pretrained/${method}${vid}_${cond_type}_torso.pth \ --aud ./pretrained/test_lm3ds.npy \ --workspace trial_test \ --bg_img ./pretrained/bc.jpg \ -O --torso --data_range 200 300

ffmpeg -y -i trial_test/results/ngp_ep0028.mp4 -i ./pretrained/test.wav -c:v copy -c:a aac trialtest/results/${method}${vid}_${cond_type}_aud.mp4

[KylinYee]

According to the execution command you gave me, there is no problem. I also saw the resulting video you provided. I think lip accuracy is also as expected. The geneface and rad-nerf we reproduced under the same conditions also have similar effects. In addition, if you want to further improve the accuracy of the synthesized lips, you can consider: 1) Comment out the post-processing of lines 135 - 158 in ./nerf/provider.py. 2) Or refer to geneface again to retrain postnet.