WillDreamer / Aurora

[NeurIPS2023] Parameter-efficient Tuning of Large-scale Multimodal Foundation Model
https://arxiv.org/abs/2305.08381
83 stars 7 forks source link

Questions about gated query transformation #4

Closed fzb408 closed 12 months ago

fzb408 commented 12 months ago

Thank you for writing such an excellent paper. Where in the code is the operation of the query gate mentioned in your paper?

xinlong-yang commented 12 months ago

Thanks for your interest. The relevant part is in 'BertLayer' class in med.py. Since you get 'layer_output' (crossattention output) and 'residual_text' (scale&shift selfattention output), you can use the following code to perform gated query transformation:

g = torch.softmax(torch.sum(torch.matmul(layer_output,residual_text.permute(0, 2, 1)),dim=-1),dim=1).unsqueeze_(2)
layer_output = layer_output * g + (1 - g) * residual_text
fzb408 commented 12 months ago

Thank you very much for your help!

------------------ 原始邮件 ------------------ 发件人: "WillDreamer/Aurora" @.>; 发送时间: 2023年11月17日(星期五) 晚上8:01 @.>; @.>;"State @.>; 主题: Re: [WillDreamer/Aurora] Questions about gated query transformation (Issue #4)

Thanks for your interest. The relevant part is in 'BertLayer' class in med.py. Since you get 'layer_output' (crossattention output) and 'residual_text' (scale&shift selfattention output), you can use the following code to perform gated query transformation: g = torch.softmax(torch.sum(torch.matmul(layer_output,residualtext.permute(0, 2, 1)),dim=-1),dim=1).unsqueeze(2) layer_output = layer_output g + (1 - g) residual_text
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you modified the open/close state.Message ID: @.***>

Arsiuuu commented 11 months ago

@fzb408 Hello~ have you reproduced successfully?

fzb408 commented 10 months ago

你好~你复制成功了吗?

您好,我没有复制成功

Arsiuuu commented 10 months ago

你好~你复制成功了吗?

您好,我没有复制成功

没有复现出结果吗,是图像还是视频

fzb408 commented 10 months ago

新一期

我想要把门限查询应用到另一个代码中没有成功,报出了loss.backward的错误