RoPE-Mixed: Improvement over Axial for n-D

lucidrains / rotary-embedding-torch

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch

MIT License

473 stars 39 forks source link

RoPE-Mixed: Improvement over Axial for n-D #25

Open tasansal opened 3 weeks ago

tasansal commented 3 weeks ago

Hi @lucidrains,

These folks talk about improving axial-RoPE performance. Some comparisons to axial-RoPE look nice, but for some, I am not convinced. I wanted to get your thoughts on this. If it makes sense, can we integrate this into the repo?

https://arxiv.org/abs/2403.13298

kabachuha commented 2 weeks ago

@tasansal you may be interested in the more recent and advanced than RoPE-mixed method I linked at #26