search

tencent-ailab / V-Express

V-Express aims to generate a talking head video under the control of a reference image, an audio, and a sequence of V-Kps images.
2.26k stars 283 forks source link

How to make blink effect? #48

Open pango99 opened 5 months ago

pango99 commented 5 months ago

below is the video I create with 5sec duration audio

https://github.com/tencent-ailab/V-Express/assets/6065571/90a4d74c-6c96-4e49-ba7d-261ff153d5a7

and this is the video I create with 30sec duration audio

https://github.com/tencent-ailab/V-Express/assets/6065571/422ea739-b648-4f78-891f-18f1f87080b9

the 5sec dura video has eye blink effect, but 30sec video haven't, why 30sec video without blink effect? How can I add blink effect to video?

tiankuan93 commented 5 months ago

This vivid blinking action comes from the change in the v-kps sequence. You're at 30s audio, but the v-kps is only 5s, so our current code interpolates the 5s v-kps to 30s by default, which attenuates the change in v-kps corresponding to blinking, which results in some of the blinking disappearing. To fix this, I suggest you use a longer natural v-kps sequence, preferably around 30s to accompany the audio, which will better preserve the facial detail of the original v-kps.

Here I provide a longer v-kps sequence corresponding to the raw video, you can replace the input v-kps file with the longer kps file I provided and do the inference again. I hope this is helpful to you.

https://github.com/tencent-ailab/V-Express/blob/c9f3d7d02eb3181e79a693eada75e12b2a3114b0/test_samples/short_case/AOC/AOC_long_kps.pth