Closed detectRecog closed 6 months ago
Can it work well with inferencing? https://github.com/PKU-YuanGroup/MoE-LLaVA?tab=readme-ov-file#cli-inference
@LinB203 Inference codes are OK. I tried "LanguageBind/MoE-LLaVA-Phi2-2.7B-4e".
Could you try use deepspeed --include localhost:0,1 moellava/train/train_mem.py
to specify 2 gpus? Or just use 1 gpu deepspeed --include localhost:0 moellava/train/train_mem.py
?
What's your deepspeed version? Is 0.9.5?
When I used one card as deepspeed --include localhost:0 moellava/train/train_mem.py
, the training begins as magic.
However, when I switched to multiple cards (2 or 4 or 8) such as "deepspeed --include localhost:0,1 moellava/train/train_mem.py", it will stuck after Formatting inputs...Skip in lazy mode
without any logs.
I test two deepspeed versions '0.9.5' and '0.13.2'. The results are the same. @LinB203
After exhaustive search on similar problems, I find that add export NCCL_P2P_DISABLE=1
can solve the problem!
However, I still wonder: 1.Why this solves the problem? 2.Does this env variable affects the training efficiency or performance?
What do you think? @LinB203
经过对类似问题的穷举查找,发现add
export NCCL_P2P_DISABLE=1
可以解决问题!但是,我仍然想知道: 1.为什么这样可以解决问题? 2.这个env变量会影响训练效率或性能吗?
你怎么认为? @LinB203
I have had this problem on other machines and solved it with this method. This environment variable is just a mode to enable multi-GPU communication as you can run on a single gpu but fail on multi-gpus. It does not affect performance.
OK, thanks a lot! Wish my experiences can help others!
Describe the issue
Issue: In pretraining or finetuning, the training always stuck after the log "Formatting inputs...Skip in lazy mode". Everytime I need to force shutting down my GPU server because it was out-of-response. Both the GUI and the SSH does not response at all.
Command (scripts/v1/phi2/pretrain.sh):
When using the zero2.json, the log stops here Log:
When using the zero2_offload.json, the log stops here Log:
I tried to remove folders under ~/.cache/torch_extensions, the stuck still happens.
My machine has 10*RTX 3090 and RAM (512G to 1T). In the above command, I just try to use 2 cards. As the machine breaks down everytime I start training, I have no idea what was wrong. Please help~
The environment exactly follows the instructions except for two additional commands "pip install deepspeed -U" and "pip install accelerate -U". I updated these two packages when I tried to solve the same problem.
Screenshots: