WillDreamer / Aurora

[NeurIPS2023] Parameter-efficient Tuning of Large-scale Multimodal Foundation Model
https://arxiv.org/abs/2305.08381
83 stars 7 forks source link

Questions about gated query transformation #4

Closed fzb408 closed 1 year ago

fzb408 commented 1 year ago

Thank you for writing such an excellent paper. Where in the code is the operation of the query gate mentioned in your paper?

xinlong-yang commented 1 year ago

Thanks for your interest. The relevant part is in 'BertLayer' class in med.py. Since you get 'layer_output' (crossattention output) and 'residual_text' (scale&shift selfattention output), you can use the following code to perform gated query transformation:

g = torch.softmax(torch.sum(torch.matmul(layer_output,residual_text.permute(0, 2, 1)),dim=-1),dim=1).unsqueeze_(2)
layer_output = layer_output * g + (1 - g) * residual_text
fzb408 commented 1 year ago

Thank you very much for your help!

------------------ 原始邮件 ------------------ 发件人: "WillDreamer/Aurora" @.>; 发送时间: 2023年11月17日(星期五) 晚上8:01 @.>; @.>;"State @.>; 主题: Re: [WillDreamer/Aurora] Questions about gated query transformation (Issue #4)

Thanks for your interest. The relevant part is in 'BertLayer' class in med.py. Since you get 'layer_output' (crossattention output) and 'residual_text' (scale&shift selfattention output), you can use the following code to perform gated query transformation: g = torch.softmax(torch.sum(torch.matmul(layer_output,residualtext.permute(0, 2, 1)),dim=-1),dim=1).unsqueeze(2) layer_output = layer_output g + (1 - g) residual_text
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: @.***>

Arsiuuu commented 1 year ago

@fzb408 Hello~ have you reproduced successfully?

fzb408 commented 10 months ago

你好~你复制成功了吗?

您好,我没有复制成功

Arsiuuu commented 10 months ago

你好~你复制成功了吗?

您好,我没有复制成功

没有复现出结果吗,是图像还是视频

fzb408 commented 10 months ago

新一期

我想要把门限查询应用到另一个代码中没有成功,报出了loss.backward的错误