Open youcaiSUN opened 2 years ago
同有这个疑问,我也跑不到原论文的效果
Hi @youcaiSUN @zhengrongz ,
Thank you so much for your interests to our work and also your careful experimental validation. It helps a lot for us to improve the quality of this repository.
I apologize for my carelessness when I reorganized the OGM-GE code. Yesterday, after correct some errors in main.py (specifically, one of the most important errors is at line 79, for 'out_v' the weight should be 'weight[:, 512:]' not 'weight[:, :512]', and the same for 'out_a'), making it consistent with our experimental code, we can obtain the correct results (higher than 0.64 with OGM-GE) in CREMA-D dataset. And the trained checkpoint has been updated, please refer to https://zenodo.org/record/6590986#.YpM4EVRBxPY. The corrected code is also updated, and you can validate again with the new code.
If you have further questions on this, please do not hesitate to contact us!
Thanks again!
Best, Andong
Hi Andong,
Thanks for your reply. Since the weight in the final fc layer can be exchangable for the audio and video features, I wonder why the relative order matters? Nevertheless, I am running the code after your fix.
Best, Licai
Hi Andong,
Thanks for your reply. Since the weight in the final fc layer can be exchangable for the audio and video features, I wonder why the relative order matters? Nevertheless, I am running the code after your fix.
Best, Licai
Actually, you can refer to ConcatFusion in model/fusion_modules.py. Before the output FC layer, x and y are concatenated in a specific order, i.e., when getting through the FC layer, each element in x or y are only multiplied with specific element in the weight of FC layer at the corresponding position. And what we want to obtain is the separate output for x or y. This is why the order matters.
I got ya. Thanks!
Got it! Thanks!
With the fixed code, I can reproduce the results in Tab. 1. Thanks again!
With the fixed code, I can reproduce the results in Tab. 1. Thanks again!
No problem! And thank you again for this issue you raised!
请问一下,有没有videoflash里的数据,原链接的数据已经损坏了,无法读入
Dear authors,
我在CREMA-D上跑了一下关于所提方法OGM-GE的消融实验,实验结果显示OGM-GE带来的提升并没有论文Table 1中的那么明显,有时甚至差于不加OGM-GE的baseline,所以很诧异希望能得到答疑解惑。视觉分支使用1张图片(同论文中保持一致)的结果如下:
其中:
可以看出:
由于实验设置方面可能并不完全相同,所以实验结果有些出入,希望能得到答疑解惑,谢谢!