search

yikaiw / CEN

[TPAMI 2023, NeurIPS 2020] Code release for "Deep Multimodal Fusion by Channel Exchanging"
MIT License
284 stars 43 forks source link

最终表示的问题 #5

Closed bourne-3 closed 3 years ago

bourne-3 commented 3 years ago

你们的工作做得非常的棒。我想问一下：经过CEN的fusion之后，如果我要进行多模态情感分析（有三个模态：文本，音频以及视频），从文章看说经过fusion之后仍然是三个输出，每个输出都有了其他模态的信息了。现在我要进行情感预测，我是只取其中的一个模态进预测还是把他们给拼接起来呢？

yikaiw commented 3 years ago

Thank you for your appreciation. Currently, our method is only verified on homogeneous modalities such as RGB, depth, normal, etc, yet is not verified on tasks containing images and texts. In our method, we adopt a learnable ensemble over the final predictions, which is empirically better than concatenating the final predictions.

bourne-3 commented 3 years ago

Ok，thanks for your reminding.I intend to use the CEN on Heterogeneous data and verify the performance of it.May I ask what the specific learnable ensemble is?Is it use each modality to make predictions and then vote?

yikaiw commented 3 years ago

The ensemble is a weighted sum of predictions, performed at Line 313-316 in models/model.py for semantic segmentation, and the weights for the ensemble are obtained by optimization.

bourne-3 commented 3 years ago

Got it，thanks.

------------------ 原始邮件 ------------------ 发件人: "yikaiw/CEN" <notifications@github.com>; 发送时间: 2021年2月28日(星期天) 中午11:05 收件人: "yikaiw/CEN"<CEN@noreply.github.com>; 抄送: "李泽彬"<595962708@qq.com>;"Author"<author@noreply.github.com>; 主题: Re: [yikaiw/CEN] 最终表示的问题 (#5)

The ensemble is a weighted sum of predictions, performed at Line 313-316 in models/model.py for semantic segmentation, and the weights for the ensemble are obtained by optimization.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.