search

vihangd / alpaca-qlora

Instruct-tune Open LLaMA / RedPajama / StableLM models on consumer hardware using QLoRA
Apache License 2.0
80 stars 11 forks source link

Lora #2

Open ghost opened 1 year ago

ghost commented 1 year ago

Hi, I noticed that you tried scaled rope. How could i use the scaled code for training a qlora with this?

vihangd commented 1 year ago

Scaled rope results were mixed. it works well on simple prompts. You can try scaled rope with finetune.py with use_rope_scaled flag. Though you will need to modify generate.py similarly