全量sft保存权重后，训练不继续

[nuoma]

在进行全量sft训练时，权重保存后（不论是by step or epoch），训练进程卡住没动静，不会继续跑下去。使用的shell： 此处没有进行do_eval，但之前的测试eval过也是同样卡住。 transformer==4.33.1

CUDA_VISIBLE_DEVICES=2,3,4,5,6,7 torchrun --nproc_per_node 6 supervised_finetuning.py \ --model_type baichuan \ --model_name_or_path /data/llm/models/baichuan/ \ --train_file_dir ../MedicalGPT-1.5.0/data/finetune/ \ --per_device_train_batch_size 2 \ --per_device_eval_batch_size 2 \ --do_train \ --use_peft False \ --num_train_epochs 3 \ --learning_rate 2e-5 \ --warmup_ratio 0.03 \ --weight_decay 0. \ --fp16 \ --logging_strategy steps \ --logging_steps 10 \ --save_steps 500 \ --save_strategy steps \ --save_total_limit 5 \ --gradient_accumulation_steps 1 \ --preprocessing_num_workers 4 \ --output_dir ./SFT3ep_FullParam \ --overwrite_output_dir \ --ddp_timeout 30000 \ --logging_first_step True \ --torch_dtype float16 \ --device_map auto \ --report_to tensorboard \ --ddp_find_unused_parameters False \ --gradient_checkpointing True \ --cache_dir ./cache \ --model_max_length 4096 \ --deepspeed deepspeed_zero_stage2_config.json \ --template_name baichuan \ --flash_attn

[nuoma]

另，企图使用--resume_from_checkpoint时提示保存目录下没有checkpoing，看了一下checkpoint文件夹里面并没有包括optimizer等文件，只有config.json config_baichuan.py generation_config.json modeling_baichuan.py pytorch_model.bin raining_args.bin

[nuoma]

更新：乱升级了一通，原问题解决了，transformers升级成4.33.2，以及accelerate

另，这个warning报的很多： warnings.warn( /data/mn/anaconda3/envs/manuo/lib/python3.9/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.

以及：/data/mn/anaconda3/envs/manuo/lib/python3.9/site-packages/deepspeed/runtime/zero/stage_1_and_2.py:1879: UserWarning: The torch.cuda.DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.) overflow_gpu = get_accelerator().ByteTensor([overflow])