jzhang38 / TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.
Apache License 2.0
7.31k stars 426 forks source link

Estimation of 2.5T Tokens Checkpoint #108

Closed mounta11n closed 7 months ago

mounta11n commented 7 months ago

Hello, do you have an estimation on when the 2.5T Tokens intermediate checkpoint will be ready?

Do you think you will achieve it before x-mas? I am very excited for next to release and can't wait heheh

Anyway, thank you so much for your work! You are contributing an extremely important part in this field!

jzhang38 commented 7 months ago

https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1195k-token-2.5T Enjoy :)