Closed r1ckya closed 5 years ago
Hi,
I'm confused about embedding in steam attention https://github.com/leaderj1001/Stand-Alone-Self-Attention/blob/485814d9a1cc443b4c502e81bbcc16ba2c74cef2/attention.py#L102
It seems that col and row emb are the same, so features are aggregated across cols and rows with the same softmax values. Isn't emb supposed to be 2 dimensional here? Like this:
emb = emb_logit_a.unsqueeze(2) + emb_logit_b.unsqueeze(1) # [m, ks, ks] p(m, a, b) emb = F.softmax(emb.view(self.m, -1), dim=-1) emb = emb.view(self.m, 1, 1, 1, 1, self.kernel_size, self.kernel_size) v_out = v_out * emb
Thanks for comments :) I'll check and fix it. Thank you for your advise !
Hi,
I'm confused about embedding in steam attention https://github.com/leaderj1001/Stand-Alone-Self-Attention/blob/485814d9a1cc443b4c502e81bbcc16ba2c74cef2/attention.py#L102
It seems that col and row emb are the same, so features are aggregated across cols and rows with the same softmax values. Isn't emb supposed to be 2 dimensional here? Like this: