search

lucidrains / rotary-embedding-torch

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch
MIT License
535 stars 43 forks source link

Does this library support 2D RoPE embeddings? #13

Closed logicchains closed 12 months ago

logicchains commented 12 months ago

The original RoPE paper suggests it can easily be extended to a 2D embedding (with two embedding dimensions rather than just one), does this library support that? If not, would it be difficult to add?

lucidrains commented 12 months ago

@logicchains yes it does, it is in the readme, under axial rotary embeddings

lucidrains commented 12 months ago

i also have an example here https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/rvt.py validated to converge well by some researchers on Eleuther