Closed DeqiangXiao closed 6 years ago
You have run out of memory on your GPU. What GPU are you using?
Oh, titan XP. That should have enough memory...
Can you run nvidia-smi
to see if you have anything else allocated on the GPU?
Thanks for your quick reply @AaronJackson Results for run nvidia-smi: Wed Nov 29 11:06:08 2017 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 381.09 Driver Version: 381.09 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 TITAN Xp Off | 0000:04:00.0 Off | N/A | | 46% 73C P2 210W / 250W | 11132MiB / 12189MiB | 95% Default | +-------------------------------+----------------------+----------------------+ | 1 TITAN Xp Off | 0000:06:00.0 Off | N/A | | 42% 67C P2 103W / 250W | 9937MiB / 12189MiB | 77% Default | +-------------------------------+----------------------+----------------------+ | 2 TITAN Xp Off | 0000:07:00.0 Off | N/A | | 53% 83C P2 172W / 250W | 11527MiB / 12189MiB | 90% Default | +-------------------------------+----------------------+----------------------+
All of your memory is allocated. I guess this is a shared machine?
Yes, this server is shared by a group. So the problem is caused by the shortage of GPU memory?
Yeah the machine is quite clearly too busy. You need to wait for some memory to become available or ask people to stop using all the GPUs. I cannot help you with this part.
OK. Thanks very much!
Hi Aaron, I downloaded your github code and successfully configured its dependences on a linux server of our lab (Ubuntu 16.04.3 LTS, 4.4.0-98-generic; Python 2.7.12; Torch7; CUDA 8.0.61; cudnn 5.1; GPU: NVIDIA TITAN Xp 12GB), but when i try to execute "run.sh", a cuda runtime error occurs:
I tried several solutions from Google, but doesn't work. Any help is appreciated!