This repository contains an educational implementation of Rotary Positional Encodings (RoPE) in PyTorch. RoPE is a method introduced in the paper RoFormer: Enhanced Transformer with Rotary Position Embedding by Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.
RoPE is a technique to incorporate positional information into the transformer architecture, specifically for self-attention mechanisms. It combines absolute and relative position information using rotation matrices, enabling the model to learn both local and global dependencies effectively.
This repository contains a PyTorch implementation of the Rotary Positional Encoding (RoPE) method. RoPE is a novel technique for positional encoding in transformers that unifies absolute and relative approaches.
Access the Colab File at: RoPE.ipynb
This implementation is for educational purposes and is based on the paper "RoFormer: Enhanced Transformer with Rotary Position Embedding" by Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Please consider referencing the paper when using or discussing this implementation.