Internal experiments - Githubissues

training-transformers-together / training-transformers-together.github.io

Contents of the main NeurIPS 2021 demo page

MIT License

2 stars 0 forks source link

Open justheuristic opened 2 years ago

justheuristic commented 2 years ago

The purpose is to ensure that training is stable. Train through warmup and see which works best by then.

[ ] d=1024, 64 layers, 5 shared groups, reversible, 8-bit lamb, batch=16k,
[ ] d=2048, 64 layers, semi-shared groups with adaptors, stable embedding, reversible, 8-bit lamb, batch=16k,

If worst comes to worst, use CollaborativeOptimizer