Open laoda513 opened 1 year ago
If I enable hybrid_engine,step3 can not train,the errors like
Traceback (most recent call last):
File "main.py", line 522, in <module>
main()
File "main.py", line 430, in main
out = trainer.generate_experience(batch_prompt['prompt'],
File "/gpt/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 98, in generate_experience
seq = self._generate_sequence(prompts, mask)
File "/gpt/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 73, in _generate_sequence
seq = self.actor_model.module.generate(prompts,
File "/gpt/DeepSpeed/deepspeed/runtime/hybrid_engine.py", line 266, in generate
self.fuse_lora_weight()
File "/gpt/DeepSpeed/deepspeed/runtime/hybrid_engine.py", line 144, in fuse_lora_weight
self._fuse_lora(self.layer_params[layer_id], self.lora_params[layer_id])
File "/gpt/DeepSpeed/deepspeed/runtime/hybrid_engine.py", line 140, in _fuse_lora
weight.data += lora_scaling * torch.matmul(lora_left_weight.t(), lora_right_weight.t())
RuntimeError: The size of tensor a (6144) must match the size of tensor b (8192) at non-singleton dimension 0
my training_scripts
ACTOR_ZERO_STAGE="--actor_zero_stage 2"
CRITIC_ZERO_STAGE="--critic_zero_stage 2"
ACTOR_MODEL_PATH=/gpt/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step1_supervised_finetuning/output/opt-1.3b-lora
CRITIC_MODEL_PATH=/gpt/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step2_reward_model_finetuning/output
OUTPUT="./output/opt-1.3b-lora"
Num_Padding_at_Beginning=1 # this is model related
Actor_Lr=5e-4
Critic_Lr=5e-6
mkdir -p $OUTPUT
deepspeed --master_port 12346 main.py \
--data_path Dahoas/rm-static \
--data_split 2,4,4 \
--actor_model_name_or_path $ACTOR_MODEL_PATH \
--critic_model_name_or_path $CRITIC_MODEL_PATH \
--num_padding_at_beginning 1 \
--per_device_train_batch_size 4 \
--per_device_mini_train_batch_size 4 \
--generation_batch_numbers 1 \
--ppo_epochs 1 \
--max_answer_seq_len 256 \
--max_prompt_seq_len 256 \
--actor_learning_rate ${Actor_Lr} \
--critic_learning_rate ${Critic_Lr} \
--num_train_epochs 1 \
--lr_scheduler_type cosine \
--gradient_accumulation_steps 1 \
--num_warmup_steps 100 \
--deepspeed --seed 1234 \
--disable_actor_dropout \
${ACTOR_ZERO_STAGE} \
${CRITIC_ZERO_STAGE} \
--actor_lora_dim 128 \
--enable_hybrid_engine \
--actor_lora_module_name decoder.layers. \
--output_dir $OUTPUT \
&> $OUTPUT/training.log
~
Same issue.
How does one disable hybrid engine?
@amaarora, to disable hybrid engine, remove _--enable_hybridengine flag from training script.
I have the same issue with you! When I do not set enable_hybrid_engine, the error is solved. However, the training speed may become slower?
But there will be a new problem. The module.generate will force the model generate a seq whose length is min_max_length which leads repeat in generation.
the issue till exists.....
When using where using hybrid engine, The output sequence always be 'a a a a ', while if I disabled hybrid engine,the output sequence is correct
here is my log with hybrid engine
the run_1.3b.sh
The log without hybrid engine:
and my device info: