search

tencent-ailab / V-Express

V-Express aims to generate a talking head video under the control of a reference image, an audio, and a sequence of V-Kps images.
2.03k stars 250 forks source link

How to produce video with lower resolution #46

Open pango99 opened 1 week ago

pango99 commented 1 week ago

the time for V-Express to produce video is too long, it takes 800sec to produce 10sec video on my RTX4070 card,I notice the output video's resolution is 512x512, I think if I reduce the resolution, the time should be shorten, so I use below command to do the test:

python inference.py ^ --reference_image_path "./test_samples/short_case/AOC/ref.jpg" ^ --audio_path ".\test_samples\short_case\10\aud.mp3" ^ --kps_path ".\test_samples\short_case\AOC\kps.pth" ^ --output_path ".\output\ztest4_no_retarget.mp4" ^ --retarget_strategy "no_retarget" ^ --num_inference_steps 25 ^ --image_width 128 ^ --image_height 128

the test result shows the time of video generation is reduced greatly, but the output video is wrong,it always display the ref image,so what's wrong with my command? Can I produce video with lower resolution?

ref_128 https://github.com/tencent-ailab/V-Express/assets/6065571/24f3b8bd-a4de-47d1-bb02-a860878e4bb8

pango99 commented 1 week ago

I had generated the kps sequence for 256x256 resolution,and use it to produce 256x256 res video use below command,

python inference.py ^ --reference_image_path "./test_samples/Res256_case/AOC/ref.jpg" ^ --audio_path "./test_samples/Res256_case/AOC/aud.mp3" ^ --kps_path ".\test_samples\Res256_case\AOC\kps.pth" ^ --output_path ".\output\ztest_Res256Case_retarget.mp4" ^ --retarget_strategy "no_retarget" ^ --num_inference_steps 25 ^ --image_height 256 ^ --image_width 256

now in the produced video, the face and mouse is moved, but the effect is not good as Res512 video, is it because 512x512 is the smallest resolution?

https://github.com/tencent-ailab/V-Express/assets/6065571/7dc5e348-f90a-4314-ac90-17601cb6ecae

tiankuan93 commented 1 week ago

We are using portrait videos of 512x512 size for our training, so the results may not be perfect for small sized faces. Maybe you can try cropping a 384x384 reference image with less background, but keeping the face size the same, as follows.

image