Closed dearleiii closed 6 years ago
same error
THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCStorage.cu line=58 error=2 : out of memory
Traceback (most recent call last):
File "save_model.py", line 36, in
Solution code:
device = torch.device("cuda:0") gpu_list = list(range(0, torch.cuda.device_count())) approximator = torch.nn.DataParallel(APXM_edsr(), device_ids=[0, 1, 2, 3, 4]) print("cuda.current_device=", torch.cuda.current_device()) print(approximator)
Data parallel across GPU is working
+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 1356 C /usr/bin/cuda_sensor 60MiB | | 0 9171 C python3 305MiB | | 0 10332 C python3 2225MiB | | 0 29879 C python3 82MiB | | 0 29955 C python3 305MiB | | 1 1356 C /usr/bin/cuda_sensor 60MiB | | 1 10140 C python3 305MiB | | 1 10332 C python3 688MiB | | 2 1356 C /usr/bin/cuda_sensor 60MiB | | 2 10332 C python3 688MiB | | 3 1356 C /usr/bin/cuda_sensor 60MiB | | 3 10332 C python3 143MiB | | 4 1356 C /usr/bin/cuda_sensor 60MiB | | 5 1356 C /usr/bin/cuda_sensor 60MiB | | 6 1356 C /usr/bin/cuda_sensor 60MiB | | 7 1356 C /usr/bin/cuda_sensor 60MiB | +-----------------------------------------------------------------------------+
leichen@gpu-compute6$ export CUDA_VISIBLE_DEVICES=0,1 leichen@gpu-compute6$ python3 save_model.py Let's use: 2 GPUs! THCudaCheck FAIL file=/pytorch/aten/src/THC/THCTensorRandom.cu line=25 error=2 : out of memory Traceback (most recent call last): File "save_model.py", line 34, in
approximator = nn.DataParallel(approximator)
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 102, in init
_check_balance(self.device_ids)
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 17, in _check_balance
dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 17, in
dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/cuda/init.py", line 290, in get_device_properties
init() # will define _get_device_properties and _CudaDeviceProperties
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/cuda/init.py", line 143, in init
_lazy_init()
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/cuda/init.py", line 161, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/THCTensorRandom.cu:25