Baichuan2-7B-Base微调报错 AttributeError: 'BaichuanConfig' object has no attribute 'z_loss_weight'AttributeError: 'BaichuanConfig' object has no attribute 'z_loss_weight'

[qingchen177]

命令： deepspeed --hostfile=$hostfile fine-tune.py \ --report_to "none" \ --data_path "data/test.json" \ --model_name_or_path "/data/models/Baichuan2-7B-Base" \ --output_dir "output" \ --model_max_length 512 \ --num_train_epochs 4 \ --per_device_train_batch_size 16 \ --gradient_accumulation_steps 1 \ --save_strategy epoch \ --learning_rate 2e-5 \ --lr_scheduler_type constant \ --adam_beta1 0.9 \ --adam_beta2 0.98 \ --adam_epsilon 1e-8 \ --max_grad_norm 1.0 \ --weight_decay 1e-4 \ --warmup_ratio 0.0 \ --logging_steps 1 \ --gradient_checkpointing True \ --deepspeed ds_config.json \ --bf16 True \ --tf32 True \ --use_lora True

报错：AttributeError: 'BaichuanConfig' object has no attribute 'z_loss_weight'AttributeError: 'BaichuanConfig' object has no attribute 'z_loss_weight'

改用Baichuan2-13B-Base直接报内存溢出： torch.cudatorch.cuda..OutOfMemoryErrorOutOfMemoryError: : CUDA out of memory. Tried to allocate 1.92 GiB (GPU 0; 23.65 GiB total capacity; 15.90 GiB already allocated; 1.04 GiB free; 21.20 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONFCUDA out of memory. Tried to allocate 1.92 GiB (GPU 1; 23.65 GiB total capacity; 15.90 GiB already allocated; 1.73 GiB free; 21.39 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

output = input.matmul(weight.t())

三张4090跑的，求解

[pandong2011]

可以参考一下 https://www.modelscope.cn/models/baichuan-inc/Baichuan2-7B-Base/feedback/prDetail/9750