fix dim error in rope - Githubissues

labmlai / annotated_deep_learning_paper_implementations

🧑‍🏫 60+ Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠

https://nn.labml.ai

MIT License

56.54k stars 5.8k forks source link

fix dim error in rope #249

Closed chen-xin-94 closed 5 months ago

chen-xin-94 commented 8 months ago

fix the following error when running _test_rotary():

""" ... x_rope = (x_rope self.cos_cached[:x.shape[0]]) + (neg_half_x self.sin_cached[:x.shape[0]]) RuntimeError: The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 3 """

vpj commented 5 months ago

Fixed it here https://github.com/labmlai/annotated_deep_learning_paper_implementations/commit/2236f6383ce66bb25f1880512a4ad0ec8f37514a