jzhang38 / TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.
Apache License 2.0
7.31k stars 426 forks source link

Loss logs #126

Closed borgr closed 6 months ago

borgr commented 6 months ago

Hey, Do you have the train\eval loss\ performance during training (I see the checkpoints ones, but wondered if you also had more frequently sampled data throughout training). Thanks

jzhang38 commented 6 months ago

We have the eval plots throughout training here: https://arxiv.org/abs/2401.02385 The training loss can be found https://wandb.ai/lance777/lightning_logs/reports/metric-train_loss-23-09-04-23-38-15---Vmlldzo1MzA4MzIw?accessToken=5eu2sndit2mo6eqls8h38sklcgfwt660ek1f2czlgtqjv2c6tida47qm1oty8ik9

borgr commented 2 months ago

Trying to extract it, are you sure there isn't a way you can save me the hassle of extracting it manually from graphs? It is mush less reliable