search

Jyonn / Legommenders

A modular recommendation system that allows the selection of different components to be combined into a new recommender.
MIT License
10 stars 1 forks source link

Question. About Attention Fusion Layer #5

Closed silent0610 closed 7 months ago

silent0610 commented 7 months ago

i have to say i am a beginner :( In paper ONCE，Attention Fusion Layer contains two formula image and image

is Attention Fusion Layer equals to AdditiveAttention module？ or is AdditiveAttention module just equals to z=Attention(Z) whether there is a linear layer before the AdditiveAttention module image i'm sorry but I can't match the code to the paper😢 can you help me understand it😊 thanks

Jyonn commented 7 months ago

Hi,

AdditiveAttention module does not include the linear layer. You can refer to line 188-189 in the model/operators/base_llm_operator.py (for BERT and LLaMA) and line 39-44 in the model/operators/attention_operator.py (for NRMS).

        outputs = self.linear(outputs)  # [B, L, D]
        outputs = self.additive_attention(outputs, mask)  # [B, D]
silent0610 commented 7 months ago

Thank you very much