Closed azton closed 1 year ago
nanoGPT is great, but it would be nice to experiment with different variations from huggingface.
Implemented GPT2 and GPTNeoX. Turns out something was going wrong in nanoGPT and these actually train in a meaningful way. (I probably messed up the changes to make FSDP work.)
nanoGPT is great, but it would be nice to experiment with different variations from huggingface.