search

leaderj1001 / Stand-Alone-Self-Attention

Implementing Stand-Alone Self-Attention in Vision Models using Pytorch
MIT License
456 stars 83 forks source link

2d embedding #3

Closed r1ckya closed 5 years ago

r1ckya commented 5 years ago

Hi,

I'm confused about embedding in steam attention https://github.com/leaderj1001/Stand-Alone-Self-Attention/blob/485814d9a1cc443b4c502e81bbcc16ba2c74cef2/attention.py#L102

It seems that col and row emb are the same, so features are aggregated across cols and rows with the same softmax values. Isn't emb supposed to be 2 dimensional here? Like this:

        emb = emb_logit_a.unsqueeze(2) + emb_logit_b.unsqueeze(1) # [m, ks, ks] p(m, a, b)
        emb = F.softmax(emb.view(self.m, -1), dim=-1)
        emb = emb.view(self.m, 1, 1, 1, 1, self.kernel_size, self.kernel_size)
        v_out = v_out * emb
leaderj1001 commented 5 years ago

Thanks for comments :) I'll check and fix it. Thank you for your advise !