search

cfoster0 / CLAP

Contrastive Language-Audio Pretraining
BSD 3-Clause "New" or "Revised" License
87 stars 4 forks source link

Add rotary position embeddings (RoPE) #12

Closed cfoster0 closed 3 years ago

cfoster0 commented 3 years ago

Should be a very easy addition. Existing JAX code for the 1D version of it is available here:

https://blog.eleuther.ai/rotary-embeddings/

cfoster0 commented 3 years ago

Working on this in the rotary branch.