Open piercus opened 3 months ago
The pretrained weights also have some strange keys, e.g. multifieldcrossatt.attention.5
, dec_blk1.attention.4
, when the model state dict does not have those keys (multifieldcrossatt only has attention.0 to 4). Was the pretrained weights be trained at a larger model architecture?
Hi @piercus ! Thank you for bringing this to our attention. There were indeed two additional linear initializations when training the Model_80.th, namely linear1 and linear2, which were not utilized in the subsequent operations. This oversight has led to the situation you've observed.
Furthermore, you are correct regarding the naming convention within the MCRM; linear1 and linear2 should indeed be renamed to linear3 and linear4 to maintain consistency throughout the codebase. We appreciate your keen eye on this matter.
Hi @htrvu ! Thank you for your careful observation regarding the pretrained weights. There were additional attention modules initialized during the training phase. These modules, however, were not utilized in the forward pass and, as a result, they were omitted from the code for the sake of simplicity and clarity. We apologize for any confusion this may have caused.
@qianyu-dlut thanks for this great work, i was having one more question regarding
MCRM
module.MCRM
naming islinear1
andlinear2
hereinf_MCRM
naming islinear3
andlinear4
hereIn
Model_80.pth
, there are 4 different layerslinear1
/linear2
/linear3
/linear4
and the values are different :
outputs
What is the difference between
linear1
andlinear3
?Thanks for your help