search

jzhang38 / TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.
Apache License 2.0
7.3k stars 425 forks source link

Pre training Continue from TinyLlama-1.1B-intermediate-step-1431k-3T #163

Closed dshah-inspird-dev closed 3 months ago

dshah-inspird-dev commented 4 months ago

I am reaching out regarding the TinyLlama-1.1B-intermediate-step-1431k-3T model checkpoint that I am currently using for my project on Hugging Face.

I have been utilizing the pretraining code for this model using TinyLlama and have successfully converted the checkpoint using the convert_hf_checkpoint.py script. Now, I am interested in continuing the pretraining process for my documents to perform domain adaptation.

However, to resume training from the existing checkpoint, I require the checkpoint file or .pt file for this specific model. I would greatly appreciate it if you could share this checkpoint file with me to facilitate the continuation of my training process.

Thank you very much for your assistance and support. Please let me know if there is any additional information or steps required from my end.

ChaosCodes commented 3 months ago

Hi, thanks for your interest! You can get the pt files from here. We will upload the early ckpts later.