An experiment to test Recurrent Memory Transformers in GPT
2
stars
1
forks
source link
readme
GPT-RMT - An experiment to test Reccurent Memory Transformers in GPT models
More info coming soon
Updates
May 5th, 2023: Trained a demo model with mixed results. Unfortunately, adding an RMT makes training incredibly slow, which slows down hyper param tuning. Will be hopefully posting results within the next two weeks