search

Alpha-VLLM / LLaMA2-Accessory

An Open-source Toolkit for LLM Development
https://llama2-accessory.readthedocs.io/
Other
2.63k stars 168 forks source link

Fine Tune LLAMA 2 on next word prediction #123

Closed bluesky314 closed 7 months ago

bluesky314 commented 7 months ago

I want to take the pretrained LLama 2 weights and fine tune them on next word prediction on my corpus of dataset. How can this be done? I dont want to instruction tune them which is what most tutorials on the web seem to be about. Kindly help me

gaopengpjlab commented 7 months ago

All examples in this repo are trained by next-word prediction.

You can start from the following demo:

https://llama2-accessory.readthedocs.io/en/latest/finetune/sg.html#single-turn-llama2-7b-on-alpaca

ChrisLiu6 commented 7 months ago

By mentioning ' I dont want to instruction tune them', I guess you mean that you don't wanna tune you model on instruction following data, but instead want to tune them on free-form corpus and to do 'continue pretraining'. If I guess right, the tutorial is https://llama2-accessory.readthedocs.io/en/latest/pretrain.html. Our pre-training implementation supports an optional pretrained_path argument https://github.com/Alpha-VLLM/LLaMA2-Accessory/blob/9f64b498940ef9c78f6f6b34ee754709c9f00d45/accessory/main_pretrain.py#L67 with which you can initialize you model for pretraining, namely continue pretraining

ChrisLiu6 commented 7 months ago

Oh this should help: https://github.com/Alpha-VLLM/LLaMA2-Accessory/issues/69#issuecomment-1720428336