Francesco215 / Language_CA

Algorithm that generates text in a way inspired by morphogenesis
9 stars 3 forks source link

Add rotary embedding to GPT2 encoder #2

Closed Francesco215 closed 1 year ago

Francesco215 commented 1 year ago

the nn.Embedding() function used to do learned positional encoding only works if the input sequence lenght is finite.

Adding rotary positional embedding will provide a way to give positional embedding to arbitrarly far away tokens

Francesco215 commented 1 year ago

This repository implements rotary embeddings but uses a different convention for the indicies of Q, K and V

Francesco215 commented 1 year ago

done #6