Finetuning using LoRA

[Nimisha-Pabbichetty]

Is it possible to finetune one of your checkpoints using LoRA on a dataset of our choice? If yes, how might we go about doing it?

[xiamengzhou]

Yes! Sheared-LLaMA models are standard huggingface models loaded with Llama model files. You can load the sheared-llama models in huggingface the same as you load standard huggingface models.

from transformers import LlamaForCausalLM
model = LlamaForCausalLM.from_pretrained("princeton-nlp/Sheared-LLaMA-1.3B")

and you can use the peft module to wrap it to be a LoRA model.

[Nimisha-Pabbichetty]

Ah thank you so much! What kind of computer specs do you use to load and train these models? I'm using 2 A100s with 80Gb memory each but keep running into cgroup out of memory errors

[xiamengzhou]

2A100 should be more than sufficient to train 1.3B and 2.7B LoRA models!

[Nimisha-Pabbichetty]

Generally how much GPU compute do you use for training?

[xiamengzhou]

I didn't run any LoRA experiments for the Sheared-LLaMA models. For getting Sheared-LLaMA models, we used full parameter tuning and we used 8 GPUs for pruning and 16 GPUs for continued pre-training.