These are primarily scripts for testing out of distribution addition.
Transformers are bad at doing addition for lengths longer than what is in their training set.
This is an exploration into how we can get transformers to actually generalize with minimal context length.
Also added some fixes were added as well to help align softmax variations with the publication, these will help with testing to which combination of configurations does best for the length generalization.
These are primarily scripts for testing out of distribution addition.
Transformers are bad at doing addition for lengths longer than what is in their training set.
This is an exploration into how we can get transformers to actually generalize with minimal context length.
Also added some fixes were added as well to help align softmax variations with the publication, these will help with testing to which combination of configurations does best for the length generalization.