search

nlpxucan / WizardLM

LLMs build upon Evol Insturct: WizardLM, WizardCoder, WizardMath
9.2k stars 713 forks source link

Have you tried LoRA? #12

Open vgoklani opened 1 year ago

vgoklani commented 1 year ago

It looks like you are doing a full fine-tuning run, have you experimented with LoRA?

xdevfaheem commented 1 year ago

I Thought That too...

shibing624 commented 1 year ago

I finetund wizardcoder with LoRA: https://github.com/shibing624/CodeAssist/blob/main/codeassist/wizard_coder.py

DarkAlchy commented 9 months ago

I can't get Oobabooga to train a LoRA for Wizard and it needs more modern code to work with what I am doing. When it thinks that 1.11.x of Pytorch is the latest version then we have an issue.

pip install accelerate==0.3.0 pip install torch===1.10.0 -f https://download.pytorch.org/whl/cu1113

That is the extent of its knowledge.

SERIOUSLY? WOW.