search

young-geng / EasyLM

Large language models (LLMs) made easy, EasyLM is a one stop solution for pre-training, finetuning, evaluating and serving LLMs in JAX/Flax.
Apache License 2.0
2.38k stars 254 forks source link

May I ask about the configs of pre-training? For example, did you use dropout? #51

Closed joytianya closed 1 year ago

joytianya commented 1 year ago

May I ask about the configs of pre-training? For example, did you use dropout?

young-geng commented 1 year ago

If you asking about OpenLLaMA, we follow the exact same configuration of LLaMA, so you can find all the hyperparameters in the LLaMA paper.

joytianya commented 1 year ago

I didn't find dropout in the paper. May I ask if llama has used dropout?