1itang
commented
10 months ago I reduced the torch version,it worked.
Closed 1itang closed 10 months ago
1itang
commented
10 months ago I reduced the torch version,it worked.
[W CUDAFunctions.cpp:108] Warning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11070). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (function operator()) Traceback (most recent call last): File "/root/.pycharm_helpers/pydev/pydevd.py", line 1483, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "/root/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/root/autodl-tmp/CMGAN/src/train.py", line 297, in
python-BaseException
mp.spawn(main, args=(world_size, args), nprocs=world_size)
File "/root/miniconda3/envs/CMGAN/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 246, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/root/miniconda3/envs/CMGAN/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 202, in start_processes
while not context.join():
File "/root/miniconda3/envs/CMGAN/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 163, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/root/miniconda3/envs/CMGAN/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
fn(i, args)
File "/root/autodl-tmp/CMGAN/src/train.py", line 279, in main
ddp_setup(rank, world_size)
File "/root/autodl-tmp/CMGAN/src/train.py", line 42, in ddp_setup
init_process_group(backend="nccl", rank=rank, world_size=world_size)
File "/root/miniconda3/envs/CMGAN/lib/python3.9/site-packages/torch/distributed/c10d_logger.py", line 74, in wrapper
func_return = func(args, **kwargs)
File "/root/miniconda3/envs/CMGAN/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1148, in init_process_group
defaultpg, = _new_process_group_helper(
File "/root/miniconda3/envs/CMGAN/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1279, in _new_process_group_helper
backend_class = ProcessGroupNCCL(backend_prefix_store, group_rank, group_size, pg_options)
RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!