search

hiyouga / LLaMA-Factory

Unified Efficient Fine-Tuning of 100+ LLMs (ACL 2024)
https://arxiv.org/abs/2403.13372
Apache License 2.0
35.17k stars 4.35k forks source link

使用Qwen-7B使用Qlora时报错在阿里的PAI-DSW #5516

Closed lzy728 closed 2 months ago

lzy728 commented 2 months ago

Reminder

System Info

Traceback (most recent call last): File "/opt/conda/bin/llamafactory-cli", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/llamafactory/cli.py", line 65, in main run_exp() File "/opt/conda/lib/python3.10/site-packages/llamafactory/train/tuner.py", line 33, in run_exp run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks) File "/opt/conda/lib/python3.10/site-packages/llamafactory/train/sft/workflow.py", line 73, in run_sft train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1623, in train return inner_training_loop( File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1960, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2911, in training_step self.accelerator.backward(loss) File "/opt/conda/lib/python3.10/site-packages/accelerate/accelerator.py", line 2123, in backward self.scaler.scale(loss).backward(**kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward torch.autograd.backward( File "/opt/conda/lib/python3.10/site-packages/torch/autograd/init.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Reproduction

CUDA_VISIBLE_DEVICES=0 llamafactory-cli train \ --stage sft \ --do_train True \ --model_name_or_path qwen/Qwen-7B \ --preprocessing_num_workers 16 \ --finetuning_type lora \ --quantization_bit 8 \ --template default \ --flash_attn auto \ --dataset_dir data \ --dataset test \ --cutoff_len 1024 \ --learning_rate 5e-05 \ --num_train_epochs 3.0 \ --max_samples 100000 \ --per_device_train_batch_size 4 \ --gradient_accumulation_steps 8 \ --lr_scheduler_type cosine \ --max_grad_norm 1.0 \ --logging_steps 5 \ --save_steps 100 \ --warmup_steps 0 \ --optim adamw_torch \ --packing False \ --report_to none \ --output_dir saves/Qwen-7B/lora/train_2024-09-23-13-57-0111111 \ --fp16 True \ --plot_loss True \ --lora_rank 8 \ --lora_alpha 16 \ --lora_dropout 0 \ --lora_target c_attn

报错Traceback (most recent call last): File "/opt/conda/bin/llamafactory-cli", line 8, in sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/llamafactory/cli.py", line 65, in main run_exp() File "/opt/conda/lib/python3.10/site-packages/llamafactory/train/tuner.py", line 33, in run_exp run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks) File "/opt/conda/lib/python3.10/site-packages/llamafactory/train/sft/workflow.py", line 73, in run_sft train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint) File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1623, in train return inner_training_loop( File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 1960, in _inner_training_loop tr_loss_step = self.training_step(model, inputs) File "/opt/conda/lib/python3.10/site-packages/transformers/trainer.py", line 2911, in training_step self.accelerator.backward(loss) File "/opt/conda/lib/python3.10/site-packages/accelerate/accelerator.py", line 2123, in backward self.scaler.scale(loss).backward(**kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/_tensor.py", line 492, in backward torch.autograd.backward( File "/opt/conda/lib/python3.10/site-packages/torch/autograd/init.py", line 251, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Expected behavior

No response

Others

No response

hiyouga commented 2 months ago

用 qwen2