Closed Wadaxiwan closed 5 months ago
Issue: When I try EP_SIZE > 1 (like EP_SIZE = 2), the program hangs at beginning. It hasn't even finished the first batch yet. May I ask if you have encountered this issue before?
Environment:
GPU: 8×A100-80G or others? 8×A100-80G Deepspeed version: 0.9.5 Torch version: 2.0.1 Transformers version: 4.37.0 Tokenizers version: 0.15.1
Command:
#!/bin/bash moe_mode="sparse" num_experts=4 top_k_experts=2 use_residual=False router_aux_loss_coef=0.01 JSON_FOLDER="ft_json" IMAGE_FOLDER="train_image_video" cd ~/jason/MoE-LLaVA HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 deepspeed moellava/train/train_mem.py \ --moe_enable True --num_experts ${num_experts} --top_k_experts ${top_k_experts} --capacity_factor 1.5 \ --moe_mode ${moe_mode} --use_residual ${use_residual} --router_aux_loss_coef ${router_aux_loss_coef} \ --train_modules fc1 fc2 wg \ --deepspeed ./scripts/zero2.json \ --model_name_or_path ./checkpoints/MoE-LLaVA-Phi2-2.7B-4e \ --version phi \ --data_path ${JSON_FOLDER}/llava_image_tune_.json ${JSON_FOLDER}/nlp_tune.json \ --image_folder ${IMAGE_FOLDER} \ --image_tower openai/clip-vit-large-patch14-336 \ --image_projector_type mlp2x_gelu \ --mm_vision_select_layer -2 \ --mm_use_im_start_end False \ --mm_use_im_patch_token False \ --image_aspect_ratio pad \ --group_by_modality_length True \ --bf16 True \ --output_dir ./checkpoints/llavaphi-2.7b-finetune-moe \ --num_train_epochs 1 \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 4 \ --gradient_accumulation_steps 2 \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 24000 \ --save_total_limit 1 \ --learning_rate 2e-5 \ --weight_decay 0. \ --warmup_ratio 0.03 \ --lr_scheduler_type "cosine" \ --logging_steps 1 \ --tf32 True \ --model_max_length 2048 \ --gradient_checkpointing True \ --dataloader_num_workers 4 \ --lazy_preprocess True \ --report_to tensorboard \ --cache_dir "./cache_dir"
Log:
No existing process group found, creating a new group named: ep_size_2 No existing process group found, creating a new group named: ep_size_2 Rank: 2 partition count [8, 8, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4] and sizes[(106024960, False), (26240, False), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (26214400, True), (102400, True)] Rank: 6 partition count [8, 8, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4] and sizes[(106024960, False), (26240, False), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (26214400, True), (102400, True)] Rank: 0 partition count [8, 8, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4] and sizes[(106024960, False), (26240, False), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (26214400, True), (102400, True)] Rank: 3 partition count [8, 8, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4] and sizes[(106024960, False), (26240, False), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (26214400, True), (102400, True)] Rank: 5 partition count [8, 8, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4] and sizes[(106024960, False), (26240, False), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (26214400, True), (102400, True)] Rank: 7 partition count [8, 8, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4] and sizes[(106024960, False), (26240, False), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (26214400, True), (102400, True)] Rank: 1 partition count [8, 8, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4] and sizes[(106024960, False), (26240, False), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (26214400, True), (102400, True)] Rank: 4 partition count [8, 8, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4] and sizes[(106024960, False), (26240, False), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (39321600, True), (26214400, True), (102400, True)] 0%| | 0/1232 [00:00<?, ?it/s]
Screenshots: You may attach screenshots if it better explains the issue.
hello, do you solve this issue? I am also trying to increase ep_size but not yet make it.
Describe the issue
Issue: When I try EP_SIZE > 1 (like EP_SIZE = 2), the program hangs at beginning. It hasn't even finished the first batch yet. May I ask if you have encountered this issue before?
Environment:
Command:
Log:
Screenshots: You may attach screenshots if it better explains the issue.