Fine-tuning of OpenKoala

[michaelfeil]

Thanks for opening up LLama!

Writing this as issue is easier said than the training itself, so propably this issue to you as authors of Koala.. As the diff weights will not be compatible to this LLama, is there plans to add the diff weights or the full-rank / standalone weights of koala?

[ttronrud]

OpenKoala diff weights will be entirely different from those for Koala itself, and would require its own fine-tuning process. Not to mention that Koala is based on LLaMA-13B. Fingers crossed that the Open-LLaMA authors start a model of that size soon, but it will take even longer to train to Chinchilla-optimal levels (~20 times the parameter count). They currently have a throughput of 2100 tok/s/TPU, which would imply ~4 TPU-years of work.

If you want something similar, on your own terms, try LoRA training Open-LLaMA yourself! The Koala authors provide their datasets, and the alpaca-lora fine-tuning procedure is still mostly up-to-date (with several exceptions). I've been doing something similar for a different task, and results have been pretty promising so far.

Koala Data: https://bair.berkeley.edu/blog/2023/04/03/koala/

Alpaca-LoRA fine-tuning procedure: https://github.com/tloen/alpaca-lora

[young-geng]

Thanks for your interest in our project. After completing the 1T token training, we are certainly interested in finetuning a chatbot on top of our model. However, we will be using a different dataset than Koala's, as the term of service of OpenAI prevents people from building models using data generated from OpenAI models to complete with them. We will probably use a combination of recently release open datasets such as Dolly and OpenAssist.

[codesoap]

FYI: There are already people finetuning OpenLLaMa to follow instructions using the databricks-dolly-15k dataset: https://github.com/yxuansu/OpenAlpaca