search

lihong2303 / AGM

[ICCV2023] The repo for "Boosting Multi-modal Model Performance with Adaptive Gradient Modulation".
MIT License
22 stars 2 forks source link

Processing of Shapley Module #9

Open Kaihui-Cheng opened 8 months ago

Kaihui-Cheng commented 8 months ago

It is a good idea to incorporate the Shapley value into balanced multi-modality learning. However, I couldn't locate the Shapley module in this project. Could you please help me find its location? Thank you.

lihong2303 commented 8 months ago

You can find it under the model initialization.

Kaihui-Cheng commented 8 months ago

Okey, I find it. Another question: in AGM Equation (1) shows $S\neq \varnothing$. But if so, in Equation (3), why there is $\phi(m_1,0^{m_2})$.

1699259082665
lihong2303 commented 8 months ago

$\phi(m{1}, 0^{m^{2}})$ is used to mute the input of modality $m{2}$. The details of the Shaply theorem can be found in "SHAPE: An Unified Approach to Evaluate the Contribution and Cooperation of Individual Modalities".