zhchaoxing
commented
3 months ago I am not sure if this is the same issue or a different one, but yeah, trying Multi-GPU SDXL finetuning on the same linux machine and got errors and can not proceed.
caching latents...
42%|████████████████████████████████████████████████████████████████████▎ | 9588/23008 [27:23<32:18, 6.92it/s][2024-05-19 06:17:40,266] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1242468 closing signal SIGTERM
[2024-05-19 06:17:40,323] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1242469 closing signal SIGTERM
[2024-05-19 06:17:40,327] torch.distributed.elastic.multiprocessing.api: [WARNING] Sending process 1242471 closing signal SIGTERM
[2024-05-19 06:17:42,066] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -9) local_rank: 2 (pid: 1242470) of binary: /home/ubuntu/train/kohya_ss_22.4.1/venv/bin/python
Traceback (most recent call last):
File "/home/ubuntu/train/kohya_ss_22.4.1/venv/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/home/ubuntu/train/kohya_ss_22.4.1/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/home/ubuntu/train/kohya_ss_22.4.1/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command
multi_gpu_launcher(args)
File "/home/ubuntu/train/kohya_ss_22.4.1/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
distrib_run.run(args)
File "/home/ubuntu/train/kohya_ss_22.4.1/venv/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
elastic_launch(
File "/home/ubuntu/train/kohya_ss_22.4.1/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/ubuntu/train/kohya_ss_22.4.1/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
========================================================
./sdxl_train.py FAILED
--------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
--------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-05-19_06:17:40
host : ubuntutrain
rank : 2 (local_rank: 2)
exitcode : -9 (pid: 1242470)
error_file: <N/A>
traceback : Signal 9 (SIGKILL) received by PID 1242470
========================================================
Issue: Multi GPU training not working ever since the "accelerate launch" has been added
Machine I have tried:
Accelerate config: It looks it does not matter, the error message is the same no mater I config it to use distributed training or not. The config I used to use to run on multi-GPU was, "distributed training -yes", "dynamo, deepspeed etc... -no", and set the correct number of GPU, and the "ALL" to use all the GPU I got. The following error message was under this config.
Version The one I`m using now is v24.0.3 But this issue has been there ever since the accelerate launch has been added
Error Message: This is from the 4x A100 machine, running on docker
The following is from the 6x RTX3090, on docker. Since it is too long that exceed the character limits, I`m pasting the last error message.
Additional info Previously before the accelerate launch was introduced to the GUI, multi-GPU was working perfectly: all you need to do was just config accelerate and it will run smoothly. So I thought it might be a good idea to ignore the accelerate launch options, such as not check the muiti-GPU checkbox. But I was wrong, it will either run on single GPU, or just error. To check whether it is an individual case, I tried different machines and OS, but the error message is very similar: it is always the “torch.distributed.elastic.multiprocessing.errors.ChildFailedError" , the only difference is the “exitcode : 1 (pid: 703)” might be other numbers
Please help