search

labmlai / annotated_deep_learning_paper_implementations

🧑‍🏫 60+ Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠
https://nn.labml.ai
MIT License
56.45k stars 5.79k forks source link

Mistake in RoPE File #256

Closed eliplutchok closed 5 months ago

eliplutchok commented 5 months ago

Hello, In this file: https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/master/labml_nn/transformers/rope/__init__.py

On line 226: rotary_pe = RotaryPositionalEmbeddings(3)

It should be instead: rotary_pe = RotaryPositionalEmbeddings(4) since the example uses 4 features. The current version of the code will return an error. Eli

Swish78 commented 5 months ago

I noticed the discrepancy as well and would be happy to help by submitting a pull request to correct it.

vpj commented 5 months ago

Thank you

Fixed it https://github.com/labmlai/annotated_deep_learning_paper_implementations/commit/2236f6383ce66bb25f1880512a4ad0ec8f37514a