PositionalEmbeddings.jl

A Julia package providing various positional embedding implementations for enriching sequence data with position information.

Features

• Absolute Positional Embeddings (AbsolutePE): Implementation of the sinusoidal position embeddings from Attention Is All You Need
• Rotary Position Embeddings (RoPE): Implementation of the rotary position embeddings from RoFormer: Enhanced Transformer with Rotary Position Embedding

Installation

using Pkg
Pkg.add("PositionalEmbeddings")

Quick Start

using PositionalEmbeddings

# Absolute Positional Embeddings
pe = AbsolutePE(512, 1024)  # embedding_size=512, max_length=1024
x = randn(Float32, 100, 512, 32)  # (seq_len, channels, batch)
x_with_pos = pe(x)

# Rotary Position Embeddings
rope = RoPE(512, 1024)  # features=512, max_length=1024
x = randn(Float32, 512, 100, 32)  # (features, seq_len, batch)
x_with_pos = rope(x)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License - see the LICENSE file for details.