Relative Position Representations

[burcehan]

Hi, Thanks for your great work! I want to know how to do the relative position representations in causal-linear model Thanks for your help

[angeloskath]

Hi,

Since the attention is never explicitly computed in any "linear" model it is not possible to use a simple relative positional encoding because it would require a different value to be added per query, making the attention computation quadratic again.

If you can come up with a factorized relative positional embedding (or if one exists already I don't know of one), then you could add it to both linear and causal-linear.

By factorized, I mean something like the following:

Let E_i be the embedding for position i then in normal positional encoding you can have something of the following form.

Q'_i = Q_i + E_i
K'_ij = K_j + E_{j-i}

which gives a dot product <Q_i, K_j> + <Ei, E{j-i}>. That second term of the dot product is what needs to be modeled in a way such that i is not present in the right hand side of the dot product. At first glance I do not see a solution.

Cheers, Angelos

[bratao]

That is a bummer, I loved the performance of the lib, and the RNN trick is amazing. @angeloskath Do you recommend a Positional Encoding that can work with fast-transformers for the Named-entity recognition in (VERY long) texts? I tried Axial (https://github.com/lucidrains/axial-positional-embedding), sinusoidal and Embedding. I got terrible results with Axial, and so-so with sinusoidal and Embedding. If you can give me any pointers, I will be very happy 😃

[angeloskath]

I can't say that I know of an absolute position embedding that would work better for long sequences. Probably a good area for research.

[angeloskath]

I am closing the issue but feel free to reopen it or open a new one, especially if you find a good way to do relative positional encoding with kernelized attention.

Thanks, Angelos

[apoorv2904]

For those interested, the proposed method in Transformers with convolutional context for ASR might be one convenient way to mimic the effects of relative positional embedding.

For ASR, this works quite well and as reported in several papers, outperforms fixed sinusoidal embeddings (with softmax transformer).

Thanks, Apoorv