GeWu-Lab / OGM-GE_CVPR2022

The repo for "Balanced Multimodal Learning via On-the-fly Gradient Modulation", CVPR 2022 (ORAL)
MIT License
236 stars 18 forks source link

Can you help me analyze the reason why my training failed when fusion_method='gated ', but my testing and evaluation failed when fusion_method='concat' #46

Open rizentan opened 5 months ago

rizentan commented 5 months ago

Hello, thank you very much for your excellence work. But there are two questions I need your help to analyze the reasons for:

(1)I encountered the following error while using the CREMAD dataset when fusionmethod='gated ': --train --dataset CREMAD --modulation OGM --fusion_method gated --modulation_starts 20 --modulationends 80 --alpha 0.8

in train_epoch
    out_v = (torch.mm(v, torch.transpose(model.module.fusion_module.fc_out.weight[:, weight_size // 2:], 0, 1))
RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x512 and 256x6)

(2)But when fusionmethod='concat ', it can be trained normally --train --dataset CREMAD --modulation OGM --fusion_method concat --modulation_starts 20 --modulationends 80 --alpha 0.8

The training was successful, but the following error occurred during the execution of the test and evaluation: _--test --dataset CREMAD --modulation OGM --fusion_method concat --modulation_starts 20 --modulationends 80 --alpha 0.8

in valid
    model.eval()
AttributeError: '_IncompatibleKeys' object has no attribute 'eval'
Trained model loaded!
Felix-fz commented 2 months ago

我遇到了相同的问题,请问您解决了吗