Sekhar-jami
commented
1 year ago Facing the same error, were you able to resolve the issue?
Open erjieyong opened 1 year ago
Sekhar-jami
commented
1 year ago Facing the same error, were you able to resolve the issue?
diogopublio
commented
1 year ago same error here any tips on fixing would be great
nishantb06
commented
1 year ago same error, possible solutions?
d4nielmeyer
commented
1 year ago Same error. Deep gratitude for any ideas
Kraegge
commented
1 year ago I have the same problem.
erjieyong
commented
1 year ago Based on the example code given, I believe that most of you would have seen the same log message wherein 0 param is trainable, which resulted in the inf error
trainable params: 0 || all params: 6755192832 || trainable%: 0.0
I manage to find another way to load and train from an existing adapter by using resume_from_checkpoint in the original alpaca_lora github.
What you need to do is 1) use the repo from alpaca_lora instead 2) Instead of adjusting any code in finetune.py, just to pass in your downloaded adapter path as part of the parameter. Example:
python finetune.py \
--base_model='decapoda-research/llama-7b-hf' \
--num_epochs=10 \
--cutoff_len=512 \
--group_by_length \
--output_dir='./lora-alpaca' \
--lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \
--lora_r=16 \
--micro_batch_size=8
--resume_from_checkpoint='./alpaca-lora-7b'
AzizovAlisher
commented
1 year ago @erjieyong not working, it starts fine-tuning LLAMA, instead of Alpaca. Any other ways?
AzizovAlisher
commented
1 year ago @erjieyong but have to point it out, it resolves the problem with 0 trainable params. Just not in a right way
seyyedaliayati
commented
1 year ago Based on the example code given, I believe that most of you would have seen the same log message wherein 0 param is trainable, which resulted in the
inf error
trainable params: 0 || all params: 6755192832 || trainable%: 0.0I manage to find another way to load and train from an existing adapter by using
resume_from_checkpointin the original alpaca_lora github.What you need to do is
- use the repo from alpaca_lora instead
- Instead of adjusting any code in finetune.py, just to pass in your downloaded adapter path as part of the parameter. Example:
python finetune.py \ --base_model='decapoda-research/llama-7b-hf' \ --num_epochs=10 \ --cutoff_len=512 \ --group_by_length \ --output_dir='./lora-alpaca' \ --lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \ --lora_r=16 \ --micro_batch_size=8 --resume_from_checkpoint='./alpaca-lora-7b'
Do you mean replacing base model with alpaca? because this command finetunes llama @erjieyong
seyyedaliayati
commented
1 year ago Based on the example code given, I believe that most of you would have seen the same log message wherein 0 param is trainable, which resulted in the
inf error
trainable params: 0 || all params: 6755192832 || trainable%: 0.0I manage to find another way to load and train from an existing adapter by using
resume_from_checkpointin the original alpaca_lora github.What you need to do is
- use the repo from alpaca_lora instead
- Instead of adjusting any code in finetune.py, just to pass in your downloaded adapter path as part of the parameter. Example:
python finetune.py \ --base_model='decapoda-research/llama-7b-hf' \ --num_epochs=10 \ --cutoff_len=512 \ --group_by_length \ --output_dir='./lora-alpaca' \ --lora_target_modules='[q_proj,k_proj,v_proj,o_proj]' \ --lora_r=16 \ --micro_batch_size=8 --resume_from_checkpoint='./alpaca-lora-7b'
Do you mean replacing base model with alpaca? because this command finetunes llama @erjieyong
d4nielmeyer
commented
1 year ago Finally I solved it by initializing the model = get_peft_model(model, config) after model = PeftModel.from_pretrained(model, LORA_WEIGHTS, torch_dtype=torch.float16) and config = LoraConfig(...). So don't comment the config. Worked quite well for me.
fredi-python
commented
1 year ago @d4nielmeyer could you make a pull request?
fredi-python
commented
1 year ago @d4nielmeyer or, please send the code here?
d4nielmeyer
commented
1 year ago Finetune.py
[...]
model = LlamaForCausalLM.from_pretrained(
base_model,
load_in_8bit=True,
torch_dtype=torch.float16,
device_map=device_map,
)
[...]
model = prepare_model_for_int8_training(model)
LORA_WEIGHTS = "tloen/alpaca-lora-7b"
model = PeftModel.from_pretrained(
model,
LORA_WEIGHTS,
torch_dtype=torch.float16,
)
config = LoraConfig(
r=lora_r,
lora_alpha=lora_alpha,
target_modules=lora_target_modules,
lora_dropout=lora_dropout,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
[...]ig = LoraConfig(...). So don't comment the config. Worked quite well for me.
@d4nielmeyer Thanks! Issue solved for me.
AhmedSSoliman
commented
1 year ago In the parameters inside LoraConfig, I think you are writing inference_mode=True. Change the inference_mode to False as in the following example:
config = LoraConfig( peft_type="LORA", r=8, lora_alpha=32, inference_mode=False, target_modules=["q_proj", "v_proj", "out_proj", "fc1", "fc2","lm_head"], lora_dropout=0.05, bias="none", task_type = "SEQ_2_SEQ_LM" )
First of all, a great thank you for posting the article and youtube video, it was very insightful!
I've tried to run the code based on your article, however i keep facing the same assertion error. Any advice?
Note that i have been trying to run your code on colab with the free gpu python version = 3.9.16 cuda version = 11.8.89
I've also noted the bug that you faced and run the same code (edited for colab) as follows: cp /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cpu.so