lucidrains / rotary-embedding-torch

Implementation of Rotary Embeddings, from the Roformer paper, in Pytorch
MIT License
535 stars 43 forks source link

Repeat order. #23

Closed AliYoussef97 closed 2 months ago

AliYoussef97 commented 5 months ago

Hello,

Thank you for the amazing work!

I had a brief question, shouldn't (n r) in repeat be (r n) here. As (r n)!=(n r), as (r n) would be equivalent to freqs = torch.cat((freqs, freqs), dim=-1). I might be compeltely wrong though, and order does not matter.

Thank you!