cupy.cuda.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory

[karunaahuja]

I want to run the training on a GPU with ID 1. So to added the argument 1 to function call [ model.to_gpu(1) ] in rrain.py . While I have ~2gb available on the GPU, when I run the network with batchsize of 32, I get the following error

cupy.cuda.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory

Here are the parameters of training model

--model models/AlexNet_flic.py \ --gpu 0 \ --epoch 1000 \ --batchsize 32 \ --snapshot 10 \ --datadir data/FLIC-full \ --channel 3 \ --flip 1 \ --size 220 \ --crop_pad_inf 1.5 \ --crop_pad_sup 2.0 \ --shift 5 \ --lcn 1 \ --joint_num 7 \

Am I doing something wrong in the way I am changing the GPU ID or is there some other problem?

Here's the full stack trace

Traceback (most recent call last): File "scripts/train.py", line 211, in model, optimizer = get_model_optimizer(args) File "scripts/train.py", line 92, in get_model_optimizer optimizer.setup(model) File "/usr/local/lib/python2.7/dist-packages/chainer/optimizer.py", line 72, in setup self.prepare() File "/usr/local/lib/python2.7/dist-packages/chainer/optimizer.py", line 87, in prepare self.init_state(param, state) File "/usr/local/lib/python2.7/dist-packages/chainer/optimizers/ada_grad.py", line 20, in init_state state['h'] = xp.zeros_like(param.data) File "/usr/local/lib/python2.7/dist-packages/cupy/creation/basic.py", line 167, in zeros_like return zeros(a.shape, dtype=dtype) File "/usr/local/lib/python2.7/dist-packages/cupy/creation/basic.py", line 144, in zeros a = empty(shape, dtype) File "/usr/local/lib/python2.7/dist-packages/cupy/creation/basic.py", line 20, in empty return cupy.ndarray(shape, dtype=dtype) File "cupy/core/core.pyx", line 87, in cupy.core.core.ndarray.init (cupy/core/core.cpp:4790) File "cupy/cuda/memory.pyx", line 252, in cupy.cuda.memory.alloc (cupy/cuda/memory.cpp:5116) File "cupy/cuda/memory.pyx", line 385, in cupy.cuda.memory.MemoryPool.malloc (cupy/cuda/memory.cpp:7701) File "cupy/cuda/memory.pyx", line 401, in cupy.cuda.memory.MemoryPool.malloc (cupy/cuda/memory.cpp:7621) File "cupy/cuda/memory.pyx", line 313, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc (cupy/cuda/memory.cpp:6556) File "cupy/cuda/memory.pyx", line 328, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc (cupy/cuda/memory.cpp:6380) File "cupy/cuda/memory.pyx", line 232, in cupy.cuda.memory._malloc (cupy/cuda/memory.cpp:5055) File "cupy/cuda/memory.pyx", line 233, in cupy.cuda.memory._malloc (cupy/cuda/memory.cpp:4970) File "cupy/cuda/memory.pyx", line 31, in cupy.cuda.memory.Memory.init (cupy/cuda/memory.cpp:1388) File "cupy/cuda/runtime.pyx", line 164, in cupy.cuda.runtime.malloc (cupy/cuda/runtime.cpp:2700) File "cupy/cuda/runtime.pyx", line 106, in cupy.cuda.runtime.check_status (cupy/cuda/runtime.cpp:1815) cupy.cuda.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory