Self-attention weights don't always sum to 1

[DennisHaijma]

Hi all,

Nice work. I have a question however. I see with in my problem setting that the self-attention (SA) weights don't always sum to ~1 (row-wise). I assume this is due to the out-of-sample approximation and structural properties of the true SA matrix. Are there ways to reduce the error and obtain a more accurate approximation or other ways to account for this? I myself will try with more landmarks and see how the dynamics change.

Thanks in advance,

Dennis

[mlpen]

Hi Dennis,

The algorithm guarantees the convergence of approximation as number of landmarks increases, and the error of approximation is bounded in terms of matrix norm.

If you want the self-attention weights to sum up 1, you can perform a renormalization after the nystrom approximation.

[DennisHaijma]

Thanks for your reply. I witness artefacts when calculating approximated weighted attention distances for each instance using its row-vector of attention weights. Whether my computation is right or wrong and given my lack of advanced linear algebra knowledge, would you suggest using the self-attention matrix for such task given the out-of-sample approximation and its implications?