search

jzhang38 / TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.
Apache License 2.0
7.3k stars 425 forks source link

Can checkpoints in the lit_gpt configuration format be open sourced? #144

Closed haiduo closed 5 months ago

haiduo commented 5 months ago

Hello author, Thank you for your open source TinyLLama1B, but I tried to train tiny_LLaMA_1b based on your pretrain codebase. After loading the huggingface format model, when I imported the lit_gpt format, I found that the stata_dict keywords of the model did not match. One was in GPT format and the other was LlamaModel, as shown below: image

Is there any good solution?
Looking forward to your reply.