search

dearleiii / PIRM-2018-SISR-Challenge

Super Resolution
https://www.pirm2018.org/PIRM-SR.html
2 stars 0 forks source link

torch.cuda.current_device() runtime error #8

Closed dearleiii closed 6 years ago

dearleiii commented 6 years ago

import torch torch.cuda.current_device() THCudaCheck FAIL file=/pytorch/aten/src/THC/THCTensorRandom.cu line=25 error=2 : out of memory Traceback (most recent call last): File "", line 1, in File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/cuda/init.py", line 332, in current_device _lazy_init() File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/cuda/init.py", line 161, in _lazy_init torch._C._cuda_init() RuntimeError: cuda runtime error (2) : out of memory at /pytorch/aten/src/THC/THCTensorRandom.cu:25

dearleiii commented 6 years ago

It looks like the GPU you picked has no memory left. Not sure which machine you are checking, but you can use something like this:

wjs@linux 4:51pm % foreach x ( seq 1 7 )
foreach? echo gpu-compute$x ; qconf -se gpu-compute$x | egrep gpuUtilization | perl -n -e '$_ =~ /^.cuda.(\d).gpuUtilization=(.),.=(.),.*$/;print "\tGPU: $1 Util: $2, Memutil: $3\n"' foreach? end gpu-compute1 GPU: 0 Util: 0, Memutil: 0 GPU: 1 Util: 0, Memutil: 0 GPU: 2 Util: 0, Memutil: 0 GPU: 3 Util: 0, Memutil: 0 GPU: 4 Util: 0, Memutil: 0 GPU: 5 Util: 0, Memutil: 0 GPU: 6 Util: 0, Memutil: 0 GPU: 7 Util: 0, Memutil: 0 gpu-compute2 GPU: 0 Util: 0, Memutil: 0 GPU: 1 Util: 0, Memutil: 0 GPU: 2 Util: 0, Memutil: 0 GPU: 3 Util: 0, Memutil: 0 GPU: 4 Util: 0, Memutil: 0 GPU: 5 Util: 0, Memutil: 0 GPU: 6 Util: 0, Memutil: 0 GPU: 7 Util: 0, Memutil: 0 gpu-compute3 ….

wjs@linux 4:52pm % foreach x ( seq 41 50 ) foreach? echo linux$x ; qconf -se linux$x | egrep gpuUtilization | perl -n -e '$_ =~ /^.cuda.(\d).gpuUtilization=(.),.=(.),.*$/;print "\tGPU: $1 Util: $2, Memutil: $3\n"' foreach? end linux41 GPU: 0 Util: 100, Memutil: 100 linux42 GPU: 0 Util: 100, Memutil: 100 linux43 GPU: 0 Util: 100, Memutil: 100 linux44 GPU: 0 Util: 0, Memutil: 0 linux45 GPU: 0 Util: 0, Memutil: 0 linux46 GPU: 0 Util: 0, Memutil: 0 linux47 GPU: 0 Util: 0, Memutil: 0 linux48 GPU: 0 Util: 0, Memutil: 0 linux49 GPU: 0 Util: 0, Memutil: 0 linux50 GPU: 0 Util: 0, Memutil: 0

Hope that helps.

Joe