tqjack
commented
1 year ago more error info was caught by adding from torch.distributed.elastic.multiprocessing.errors import record
Fatal Python error: Bus error
Thread 0x00007f1ac3b25740 (most recent call first): File "/usr/local/lib/python3.8/dist-packages/bmtrain/nccl/init.py", line 77 in commInitRank File "/usr/local/lib/python3.8/dist-packages/bmtrain/init.py", line 99 in init_distributed File "train.py", line 9 in main File "/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346 in wrapper File "train.py", line 129 in module
As mentioned, try to run the example provided in example folder by using run.sh script. But throw me this
WARNING:torch.distributed.run:
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 37831) of binary: /usr/local/bin/python Traceback (most recent call last): File "/usr/local/bin/torchrun", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 762, in main
run(args)
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 753, in run
elastic_launch(
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 132, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
train.py FAILED
Failures: [1]: time : 2023-03-21_01:00:08 host : 2b0ea7cdb636 rank : 1 (local_rank: 1) exitcode : -7 (pid: 37832) error_file: <N/A> traceback : Signal 7 (SIGBUS) received by PID 37832 [2]: time : 2023-03-21_01:00:08 host : 2b0ea7cdb636 rank : 2 (local_rank: 2) exitcode : -7 (pid: 37833) error_file: <N/A> traceback : Signal 7 (SIGBUS) received by PID 37833 [3]: time : 2023-03-21_01:00:08 host : 2b0ea7cdb636 rank : 3 (local_rank: 3) exitcode : -7 (pid: 37834) error_file: <N/A> traceback : Signal 7 (SIGBUS) received by PID 37834
Root Cause (first observed failure): [0]: time : 2023-03-21_01:00:08 host : 2b0ea7cdb636 rank : 0 (local_rank: 0) exitcode : -7 (pid: 37831) error_file: <N/A> traceback : Signal 7 (SIGBUS) received by PID 37831
I am using a server with 8*3090
630GB cpu RAM python 3.8 cuda 11.8