search

leaderj1001 / Stand-Alone-Self-Attention

Implementing Stand-Alone Self-Attention in Vision Models using Pytorch
MIT License
456 stars 83 forks source link

Add `sum(dim=2)` for dot-product #24

Closed jnhwkim closed 3 years ago

jnhwkim commented 3 years ago

I believe .sum(dim=2) should be added to the below line:

https://github.com/leaderj1001/Stand-Alone-Self-Attention/blob/a983f0f643632b1f2b7b8b27693182f22e9e574c/attention.py#L48

for dot-product in Equation 2 in the paper.

jnhwkim commented 3 years ago

Duplicated with #10.