[WIP] fineweb10B/gpt2 recipe, and supporting changes - Githubissues

eole-nlp / eole

Open language modeling toolkit based on PyTorch

https://eole-nlp.github.io/eole

MIT License

24 stars 6 forks source link

[WIP] fineweb10B/gpt2 recipe, and supporting changes #32

Open francoishernandez opened 1 month ago

francoishernandez commented 1 month ago

This is blatantly inspired from https://github.com/karpathy/llm.c/discussions/481.

Some changes in the process:

support adamw optimizer;
support cosine style decay (linear warmup + cosine decrease to 0);
removed param_init_glorot in favor of a more generic param_init_method;
check all recipes config in tests.

TODO:

[ ] proper README
[ ] full reproduction run / multi-gpu run + logs/charts
[x] hellaswag benchmark (#38)