Closed Francesco215 closed 1 year ago
the nn.Embedding() function used to do learned positional encoding only works if the input sequence lenght is finite.
nn.Embedding()
Adding rotary positional embedding will provide a way to give positional embedding to arbitrarly far away tokens
This repository implements rotary embeddings but uses a different convention for the indicies of Q, K and V
done #6
the
nn.Embedding()
function used to do learned positional encoding only works if the input sequence lenght is finite.Adding rotary positional embedding will provide a way to give positional embedding to arbitrarly far away tokens