How to use this on custom videos

zhangzjn / APB2FaceV2

An improved version of APB2Face: Real-Time Audio-Guided Multi-Face Reenactment

82 stars 13 forks source link

How to use this on custom videos #7

Closed niladridutt closed 3 years ago

niladridutt commented 3 years ago

Right now the dataloader can only parse ANNVii dataset it seems. How can I use it on custom videos? How to get the .t7 file?

zhangzjn commented 3 years ago

You need to get synchronized image, audio, and landmark data for your own dataset. Details to generate the .t7 file can be found in data/AnnScripts/script_AnnVI.py.

niladridutt commented 3 years ago

Thank you so much for your reply and uploading the scripts! What do you mean by "synchronized audio, image, pose, landmark, and blink information" in the paper? And how do I get it (for example dlib 68 shape predictor?)? It'll be really helpful if you could explain what is synced images. Thanks!

zhangzjn commented 3 years ago

Synchronization means that the image and audio have the same timestamp, so that the audio can be used to control the facial movement.
For the landmark detector, you can refer to https://github.com/davisking/dlib or https://www.faceplusplus.com.cn/face-detection

niladridutt commented 3 years ago

Thanks! Do you have any script that generates the landmarks of the images and cropped images in the format needed by script_AnnVI.py ? It'll be very helpful if you could upload them as well. Essentially if you have a script that generates landmark files from the video it'll be great

zhangzjn commented 3 years ago

I'm sorry, but you can refer to str_landmark to create the custom dataset that matches script_AnnVI.py.