ROFORMER: ENHANCED TRANSFORMER WITH ROTARY POSITION EMBEDDING

[5g4s]

https://arxiv.org/abs/2104.09864 https://blog.eleuther.ai/rotary-embeddings/

[5g4s]

Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. RoPE can leverage positional information into the learning process.

Problem

Absolute positional encoding is not always, particularly meaningful due to the common practices of short sentences and phrases together in a single context and breaking up sentences across contexts.

[5g4s]

The existing method directly adds the position information to the context representations. Unlikely, our approach aims to derive the relative position encoding by incorporating relative position information with the rotation of context representations.

[5g4s]

[5g4s]

Our method encourages faster convergence.