Chat v1.0 training recipe

jzhang38 / TinyLlama

The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

Apache License 2.0

7.31k stars 426 forks source link

Chat v1.0 training recipe #137

Closed normster closed 5 months ago

normster commented 5 months ago

Hi,

In the model card for the chat v1.0 model, it mentions following the "Zephyr training recipe". Does this mean using the alignment-handbook codebase or another reproduction of the Zephyr recipe?

Thanks!

jzhang38 commented 5 months ago

Yes, we use the alignment handbook without changing any hyperparam.