Closed silent0610 closed 7 months ago
Hi,
AdditiveAttention module does not include the linear layer. You can refer to line 188-189 in the model/operators/base_llm_operator.py
(for BERT and LLaMA) and line 39-44 in the model/operators/attention_operator.py
(for NRMS).
outputs = self.linear(outputs) # [B, L, D]
outputs = self.additive_attention(outputs, mask) # [B, D]
Thank you very much
i have to say i am a beginner :( In paper ONCE,Attention Fusion Layer contains two formula
and
![image](https://github.com/Jyonn/Legommenders/assets/130958660/eb9b4834-4274-4a24-b78c-b8913f33d966)
is Attention Fusion Layer equals to AdditiveAttention module? or is AdditiveAttention module just equals to
i'm sorry but I can't match the code to the paper😢
can you help me understand it😊
thanks
z=Attention(Z)
whether there is a linear layer before the AdditiveAttention module