search

lancopku / label-words-are-anchors

Repository for Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning
MIT License
144 stars 12 forks source link

Problem about S_{wq} computation #25

Closed lucky9-cyou closed 4 months ago

lucky9-cyou commented 4 months ago

Original Code: https://github.com/lancopku/label-words-are-anchors/blob/74b58040999c23285e4a1d221309398f14b0fe57/attention_attr.py#L117

I think it should be 

proportion1 = 0
for class_pos in class_poss:
    proportion1 += saliency[class_pos, :class_pos].sum()

according to sum operation in following code: https://github.com/lancopku/label-words-are-anchors/blob/74b58040999c23285e4a1d221309398f14b0fe57/attention_attr.py#L123

leanwang326 commented 4 months ago

啊是的，不过对gpt而言它的attention是个三角阵，所以剩下的部分saliency也是0，所以写成那样了（写成saliency[class_poss, :class_poss]在class_poss是个列表的时候稍微有点麻烦就是了）

lucky9-cyou commented 4 months ago

啊是的，不过对gpt而言它的attention是个三角阵，所以剩下的部分saliency也是0，所以写成那样了（写成saliency[class_poss, :class_poss]在class_poss是个列表的时候稍微有点麻烦就是了）

Got it. Thanks for your reply.