when `per_device_eval_batch_size` > 1 and launch by deepspeed, RuntimeError: Tensors must be contiguous

[jiahuanluo]

RuntimeError: Tensors must be contiguous occurs when per_device_eval_batch_size > 1 cmd: deepspeed --include localhost:0,1,2,3,4,5,6,7 --master_port $MASTER_PORT src/train_bash.py \ --stage sft \ --model_name_or_path THUDM/chatglm2-6b \ --checkpoint_dir ${CHECKPOINT} \ --do_predict \ --dataset dev_data\ --overwrite_cache \ --finetuning_type lora \ --output_dir ${CHECKPOINT}/predict \ --overwrite_cache \ --per_device_eval_batch_size 4 \ --max_source_length 1024 \ --max_target_length 128 \ --max_samples 1000 \ --predict_with_generate \ --plot_loss \ --fp16

[kuailehaha]

我在ChatGLM和LLaMA的efficient_tuning下,用deepspeed和accelerate的多卡infer同报错 accelerate launch \ ./LLaMA-Efficient-Tuning/src/train_bash.py \ --max_samples 50 \ --model_name_or_path "Llama-2-13B-fp16/" \ --do_predict \ --dataset alpaca_zh \ --dataset_dir "LLaMA-Efficient-Tuning/data"\ --finetuning_type lora \ --output_dir Efficient_Tuning/llama2-13b \ --per_device_eval_batch_size 4 \ --predict_with_generate \ --fp16 蹲解决方法

[hiyouga]

目前 do_predict 仅支持单卡

[jiahuanluo]

LLaMA-Efficient-Tuning 之前是可以的，代码越更新越多bug了

[kuailehaha]

是这样的 6月份的版本可以accelerate launch多卡

[hiyouga]

@kuailehaha pull 最新的代码试一下