The lip shakes so much

[clumsynope]

@Weizhi-Zhong Thanks for your extraordinary work. But when I run the code, I found the lip shakes so much. There may be lack of some postprocess? The result as following: https://drive.google.com/file/d/1L6XfTZKV_nvqv1FW3fFp1-9quOUgALYe/view?usp=sharing

[Weizhi-Zhong]

Hi, thanks for your interest. As depicted in the paper, the only post-processing is used for generated faces through a Gaussian-smoothed face mask. And the result generated by the original code will not shake. As for your result, I think maybe you modified the code and introduced some errors. Or do you install the correct version package of the requirements.txt?

It seems the problems occur in the audio-to-landmark stage.

[clumsynope]

I did not modify the code. Run the following command and installation tutorial as you mentioned in the README: CUDA_VISIBLE_DEVICES=0 python inference_single.py --input ./test/template_video/129.mp4 --audio ./test/template_video/audio2.wav @Weizhi-Zhong

I'm so confused

[Weizhi-Zhong]

I did not modify the code. Run the following command and installation tutorial as you mentioned in the README: CUDA_VISIBLE_DEVICES=0 python inference_single.py --input ./test/template_video/129.mp4 --audio ./test/template_video/audio2.wav @Weizhi-Zhong

I'm so confused do you install the correct version of torch and so on?

[clumsynope]

I did not modify the code. Run the following command and installation tutorial as you mentioned in the README: CUDA_VISIBLE_DEVICES=0 python inference_single.py --input ./test/template_video/129.mp4 --audio ./test/template_video/audio2.wav @Weizhi-Zhong I'm so confused do you install the correct version of torch and so on?

Yes, torch==1.10.0 torchvision==0.11.0 @Weizhi-Zhong Can you please run the command: CUDA_VISIBLE_DEVICES=0 python inference_single.py --input ./test/template_video/129.mp4 --audio ./test/template_video/audio2.wav check whether has the same problem.

[Weizhi-Zhong]

Hi~, I ran the command "CUDA_VISIBLE_DEVICES=0 python inference_single.py" as described in README.md, and I got the result on my machine with gpu RTX3090 like: https://1drv.ms/v/s!Amqu9u09qiUGi7coEZFOuiL9P9xx_Q?e=wOwMDs Besides torch==1.10.0 torchvision==0.11.0, the version of other packages may affect the result. Please refer to requirement.txt. Thanks

[Weizhi-Zhong]

--audio ./test/template_video/audio2.wav

Hi~, from your filename of GoogleDrive:"666result_N_25_Nl_15.mp4", it seems that your input_video_path is set to 666.mp4. Refer to the following code in inference_single.py outfile_path = os.path.join(output_dir,'{}N{}Nl{}.mp4'.format(input_video_path.split('/')[-1][:-4] + 'result', ref_img_N, Nl))

But 666.mp4 is the file on my machine that is too large to be uploaded to GitHub, Sorry~

Just now, I update the inference_single.py, modifying the default path to the following: parser.add_argument('--input', '--input_template_video', type=str, default='./test/template_video/129.mp4')

[Weizhi-Zhong]

--audio ./test/template_video/audio2.wav

Hi~, from your filename of GoogleDrive:"666result_N_25_Nl_15.mp4", it seems that your input_video_path is set to 666.mp4. Refer to the following code in inference_single.py outfile_path = os.path.join(output_dir,'{}N{}Nl{}.mp4'.format(input_video_path.split('/')[-1][:-4] + 'result', ref_img_N, Nl))

But 666.mp4 is the file on my machine that is too large to be uploaded to GitHub, Sorry~

Just now, I update the inference_single.py, modifying the default path to the following: parser.add_argument('--input', '--input_template_video', type=str, default='./test/template_video/129.mp4')

@clumsynope , you can try with the new code of inference_single.py

[clumsynope]

ok, I try it now.

[Weizhi-Zhong]

ok, I try it now.

Hi~, does it work?

[clumsynope]

ok, I try it now.

Hi~, does it work?

Sorry for replying to you late. The problem is fixed.