YuanxunLu / LiveSpeechPortraits

Live Speech Portraits: Real-Time Photorealistic Talking-Head Animation (SIGGRAPH Asia 2021)
MIT License
1.16k stars 200 forks source link

i want to try to train a model of personal speaker #44

Closed VERMANs closed 2 years ago

VERMANs commented 2 years ago

Hi , i want to try to train a model of personal speaker. Can I train only the Audio2Feature model and Audio2Headpose model based on your APC model weights?(In other words, is the APC model generalizable?) Can you give me some advice, thanks a lot

YuanxunLu commented 2 years ago

APC model is a speaker-independent model (in my setting, all persons use the same APC model if you check the codes), so you can just use the pre-trained weights to extract deep audio features.

You can also train a new APC model or other deep audio feature extraction network to extract deep audio features.

VERMANs commented 2 years ago

APC model is a speaker-independent model (in my setting, all persons use the same APC model if you check the codes), so you can just use the pre-trained weights to extract deep audio features.

You can also train a new APC model or other deep audio feature extraction network to extract deep audio features.

Thanks for your reply, great, I noticed the "APC_epoch_160.model" file. Another problem here is that APC_feature_base.npy is stored in each different role directory. I printed out their values and found that they are not equal. I looked up the code and found out that it works on "manifold projection", can you tell me how they are created?

YuanxunLu commented 2 years ago

APC_feature_base.npy contains the APC features of training data. Please check it in the paper. APC model can be found in the code, and you can use it to generate APC features for your audio.

VERMANs commented 2 years ago

ok,thanks,I'm going to see your paper.Cheers for your generosity!