Finetune issue

[white2018]

@Yuliang-Liu Nice work! I run into finetune issue as follows

2 GPUs of NVIDIA A800 80G is employed during training. The finetune script looks like CUDA_VISIBLE_DEVICES=$gpu torchrun \ --nnodes=1 \ --node_rank=0 \ --master_addr=0.0.0.0 \ --nproc_per_node=${GPUS} \ --master_port=${MASTER_PORT} \ internvl/train/internvl_chat_finetune.py \ --model_name_or_path "models/OpenGVLab/InternVL2-2B" \ --conv_style "internlm2-chat" \ --output_dir ${OUTPUT_DIR} \ --meta_path "shell/data/train-finetune.json" \ --overwrite_output_dir True \ --force_image_size 448 \ --max_dynamic_patch 6 \ --down_sample_ratio 0.5 \ --drop_path_rate 0.0 \ --freeze_llm True \ --freeze_mlp True \ --freeze_backbone True \ --use_llm_lora 16 \ --vision_select_layer -1 \ --dataloader_num_workers 4 \ --bf16 True \ --num_train_epochs 1 \ --per_device_train_batch_size ${PER_DEVICE_BATCH_SIZE} \ --gradient_accumulation_steps ${GRADIENT_ACC} \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 200 \ --save_total_limit 1 \ --learning_rate 4e-6 \ --weight_decay 0.01 \ --warmup_ratio 0.03 \ --lr_scheduler_type "cosine" \ --logging_steps 1 \ --max_seq_length 4096 \ --do_train True \ --grad_checkpoint True \ --group_by_length True \ --dynamic_image_size True \ --use_thumbnail True \ --ps_version 'v2' \ --deepspeed "zero_stage1_config.json" \ --report_to "tensorboard"

Could you pls give me some clues to fix it? Thanks a lot

[mxin262]

Hi~ The default is to run with eight GPUs. If you use two GPUs, you need to set --nproc_per_node to 2.

[white2018]

Hi~ The default is to run with eight GPUs. If you use two GPUs, you need to set --nproc_per_node to 2.

The training commands looks like: ++ CUDA_VISIBLE_DEVICES=0,1 ++ torchrun --nnodes=1 --node_rank=0 --master_addr=0.0.0.0 --nproc_per_node=2 --master_port=20135 internvl/train/internvl_chat_finetune.py --model_name_or_path models/OpenGVLab/InternVL2-2B --conv_style internlm2-chat --output_dir minimonkey_chat_lora --meta_path shell/data/train-finetune.json --overwrite_output_dir True --force_image_size 448 --max_dynamic_patch 6 --down_sample_ratio 0.5 --drop_path_rate 0.0 --freeze_llm True --freeze_mlp True --freeze_backbone True --use_llm_lora 16 --vision_select_layer -1 --dataloader_num_workers 4 --bf16 True --num_train_epochs 1 --per_device_train_batch_size 2 --gradient_accumulation_steps 4 --evaluation_strategy no --save_strategy steps --save_steps 200 --save_total_limit 1 --learning_rate 4e-6 --weight_decay 0.01 --warmup_ratio 0.03 --lr_scheduler_type cosine --logging_steps 1 --max_seq_length 4096 --do_train True --grad_checkpoint True --group_by_length True --dynamic_image_size True --use_thumbnail True --ps_version v2 --deepspeed zero_stage1_config.json --report_to tensorboard

which leads to core, the snapshot is as follow:

[mxin262]

Do you encounter this issue when using zero_stage3_config.json?