search

microsoft / Swin-Transformer

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows".
https://arxiv.org/abs/2103.14030
MIT License
13.98k stars 2.06k forks source link

Where is the dot product with the query vector for the relative position index #348

Closed swarajnanda2021 closed 9 months ago

swarajnanda2021 commented 9 months ago

Per the following code: attn = attn + relative_position_bias.unsqueeze(0), found in the windowed attention operation, is there a dot product operation with the query vector that is needed per the Vaswani et al. (2018) paper (https://arxiv.org/pdf/1803.02155.pdf), equation 5, second term on the numerator?

swarajnanda2021 commented 9 months ago

Never mind, I understand the implementation now. This is the revised, faster version.