During an initial training run of SmolGPT, the model seemed to train for a bit and then plateau. During evaluation, I found that, regardless of what the model was fed, it would just output "MMMMMMMMMMMMM" repeatedly (i guess it was hungry?). There are several standard debugging steps that will need to be followed to figure out what is going wrong and these should be added to tricycle. Some include:
Storing sample evaluations during experiments
Checking that loss falls to 0 on a single batch
I am sure that more techniques will be used when I actually start looking into it and i'll add then in this branch accordingly
During an initial training run of SmolGPT, the model seemed to train for a bit and then plateau. During evaluation, I found that, regardless of what the model was fed, it would just output "MMMMMMMMMMMMM" repeatedly (i guess it was hungry?). There are several standard debugging steps that will need to be followed to figure out what is going wrong and these should be added to tricycle. Some include: