search

gordicaleksa / pytorch-original-transformer

My implementation of the original transformer model (Vaswani et al.). I've additionally included the playground.py file for visualizing otherwise seemingly hard concepts. Currently included IWSLT pretrained models.
https://youtube.com/c/TheAIEpiphany
MIT License
983 stars 169 forks source link

Frequency in the positional encodings #5

Open FAhtisham opened 3 years ago

FAhtisham commented 3 years ago

What does the frequency represent in positional encoding ? Why do we need to multiply it with the positional values?

frequencies = torch.pow(10000., -torch.arange(0, model_dimension, 2, dtype=torch.float) / model_dimension)