search

kyegomez / LongNet

Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"
https://discord.gg/qUtxnK2NMf
Apache License 2.0
663 stars 63 forks source link

KeyError: 'module.token_embs.0.gamma' #9

Closed pokameng closed 11 months ago

pokameng commented 11 months ago

@kyegomez hello bro!! Thanks for your code When i run training.py , something is wrong: File "LongNet/training.py", line 364, in decoupled_optimizer no_decay_param.append(param_dict[param]) KeyError: 'module.token_embs.0.gamma'

kyegomez commented 11 months ago

@pokameng Hey 👋 Thank you for being a Human 😊

And, yeah we're still working on the model and the training script! It'll be done soon!