search

yiranran / Audio-driven-TalkingFace-HeadPose

Code for "Audio-driven Talking Face Video Generation with Learning-based Personalized Head Pose" (Arxiv 2020) and "Predicting Personalized Head Movement From Short Video and Speech Signal" (TMM 2022)
https://ieeexplore.ieee.org/document/9894719
718 stars 147 forks source link

Some questions about the dataset of LRW when training #21

Open Quan233 opened 4 years ago

Quan233 commented 4 years ago

Hi, thanks for the great job. I have a little question about the training process of the audio to epression and pose mapping part. While testing there is a finetune process, and I found it in the dataset.py which the class " News_1D_lstm_3dmm_pose" is selected.

I guess when trained with LRW dataset, the corresponding dataset should be "LRW_1D_lstm_3dmm_pose", it is correct or not ?

If it is correct, I saw it there is a random index from here : "r = random.choice([x for x in range(3,8])", seems like the mfcc features and expression features are random sampled from time. So when training , for one sample the features may be not equal because of this random selection? I just wondering whether will it bring some problems while training ?

Thank you!

yiranran commented 4 years ago

Yes this part is random, we randomly take 16 frames from each 29 frame clip in LRW dataset during training. We have not tested the influence of this randomness.. Thank you for reminding.

yangchunyong commented 3 years ago

when use my own data for training，can I drop the random part, and use whole frames of the video sample to train?Thanks for your project and answer.

ghost commented 3 years ago

@yangchunyong have you sloved this problem? how do you do when use your own data for training， drop the random part, and use whole frames of the video sample to train.

MitchellX commented 3 years ago

hi~ @yangchunyong and @chenbolinstudent could you please share me with the LRW dataset? I have applied for permission to use this dataset, but the administrator is on vacation now, quote by Rob Cooper "Thanks for your message. I'm on holiday, back on Thursday 7th January. I'll get back to you then."

I'm in a hurry to use it, could you please content me by email: xiangmingcan@iie.ac.cn. Thank you very much.