linjieli222
commented
3 years ago @Mao-JianGuo
Thanks for your interests in our project and sorry for the late response.
We did not pre-train HERO with videoQA head, so the following weights are not initialized:
['qa_pool.weight', 'qa_pred_head.linear_1.weight', 'qa_pred_head.linear_1.bias', 'qa_pred_head.LayerNorm.weight', 'qa_pred_head.LayerNorm.bias', 'qa_pred_head.linear_2.weight', 'qa_pred_head.linear_2.bias', 'st_ed_pool.weight',
'st_ed_pred_head.linear_1.weight', 'st_ed_pred_head.linear_1.bias', 'st_ed_pred_head.LayerNorm.weight',
'st_ed_pred_head.LayerNorm.bias', 'st_ed_pred_head.linear_2.weight', 'st_ed_pred_head.linear_2.bias']The following weights are used in VSM pre-training task, but not used in VideoQA
['q_feat_attn.query_input_proj.LayerNorm.weight', 'q_feat_attn.query_input_proj.LayerNorm.bias',
'q_feat_attn.query_input_proj.net.1.weight', 'q_feat_attn.query_input_proj.net.1.bias',
'q_feat_attn.query_pos_embed.position_embeddings.weight', 'q_feat_attn.query_pos_embed.LayerNorm.weight',
'q_feat_attn.query_pos_embed.LayerNorm.bias', 'q_feat_attn.query_self_attention.self.query.weight',
'q_feat_attn.query_self_attention.self.query.bias', 'q_feat_attn.query_self_attention.self.key.weight',
'q_feat_attn.query_self_attention.self.key.bias', 'q_feat_attn.query_self_attention.self.value.weight',
'q_feat_attn.query_self_attention.self.value.bias', 'q_feat_attn.query_self_attention.output.dense.weight',
'q_feat_attn.query_self_attention.output.dense.bias', 'q_feat_attn.query_self_attention.output.LayerNorm.weight',
'q_feat_attn.query_self_attention.output.LayerNorm.bias', 'q_feat_attn.modular_vector_mapping.weight',
'video_query_linear.weight', 'video_query_linear.bias', 'video_st_predictor.weight', 'video_ed_predictor.weight',
'vocab_padded']
Hello, I'm running the train_videoQA.py , the program reports an error and shows that the model parameters are missing. What's the matter?
04/28/2021 02:00:52 - INFO - model.modeling_utils - Weights of HeroForVideoQA not initialized from pretrained model: ['qa_pool.weight', 'qa_pred_head.linear_1.weight', 'qa_pred_head.linear_1.bias', 'qa_pred_head.LayerNorm.weight', 'qa_pred_head.LayerNorm.bias', 'qa_pred_head.linear_2.weight', 'qa_pred_head.linear_2.bias', 'st_ed_pool.weight', 'st_ed_pred_head.linear_1.weight', 'st_ed_pred_head.linear_1.bias', 'st_ed_pred_head.LayerNorm.weight', 'st_ed_pred_head.LayerNorm.bias', 'st_ed_pred_head.linear_2.weight', 'st_ed_pred_head.linear_2.bias'] 04/28/2021 02:00:52 - INFO - model.modeling_utils - Weights from pretrained model not used in HeroForVideoQA: ['q_feat_attn.query_input_proj.LayerNorm.weight', 'q_feat_attn.query_input_proj.LayerNorm.bias', 'q_feat_attn.query_input_proj.net.1.weight', 'q_feat_attn.query_input_proj.net.1.bias', 'q_feat_attn.query_pos_embed.position_embeddings.weight', 'q_feat_attn.query_pos_embed.LayerNorm.weight', 'q_feat_attn.query_pos_embed.LayerNorm.bias', 'q_feat_attn.query_self_attention.self.query.weight', 'q_feat_attn.query_self_attention.self.query.bias', 'q_feat_attn.query_self_attention.self.key.weight', 'q_feat_attn.query_self_attention.self.key.bias', 'q_feat_attn.query_self_attention.self.value.weight', 'q_feat_attn.query_self_attention.self.value.bias', 'q_feat_attn.query_self_attention.output.dense.weight', 'q_feat_attn.query_self_attention.output.dense.bias', 'q_feat_attn.query_self_attention.output.LayerNorm.weight', 'q_feat_attn.query_self_attention.output.LayerNorm.bias', 'q_feat_attn.modular_vector_mapping.weight', 'video_query_linear.weight', 'video_query_linear.bias', 'video_st_predictor.weight', 'video_ed_predictor.weight', 'vocab_padded'] Selected optimization level O2: FP16 training with FP32 batchnorm and FP32 master weights.