GpuArrayException: b'cuMemAlloc: CUDA_ERROR_OUT_OF_MEMORY: out of memory'

[raghavmenon]

Hi,

I have a NVIDIA GTX 1080 GPU. I had been trying to implement Capsule net using nolearn, Lasagne and Theano but have been receiving the following error. Any help is appreciated.

Layer information

name size

---

0 1x36x60 1 50x28x52 2 164x10x22 3 41x4 4 41x4 5 41x4 6 41

Traceback (most recent call last):

File "", line 1, in runfile('/home/raghav_menon/ZeroResource/SABC/Data_manip_Caps_net.py', wdir='/home/raghav_menon/ZeroResource/SABC')

File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 705, in runfile execfile(filename, namespace)

File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/spyder/utils/site/sitecustomize.py", line 102, in execfile exec(compile(f.read(), filename, 'exec'), namespace)

File "/home/raghav_menon/ZeroResource/SABC/Data_manip_Caps_net.py", line 153, in caps_nn = Caps_nn(X,y_train_int,frame_length,feat_ndims,Epoch) # Training Input without normalisation

File "/home/raghav_menon/ZeroResource/SABC/Capsnet.py", line 224, in Caps_nn caps_nn = Caps_NN(Epoches,frame_length,feat_ndims,X_train.shape[0],y_train).fit(X_train, y_train)

File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/nolearn/lasagne/base.py", line 693, in fit self.train_loop(X, y, epochs=epochs)

File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/nolearn/lasagne/base.py", line 756, in train_loop self.apply_batch_func(self.trainiter, Xb, yb))

File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/nolearn/lasagne/base.py", line 847, in apply_batch_func return func(Xb) if yb is None else func(Xb, yb)

File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 917, in call storage_map=getattr(self.fn, 'storage_map', None))

File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/theano/gof/link.py", line 325, in raise_with_op reraise(exc_type, exc_value, exc_trace)

File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/six.py", line 692, in reraise raise value.with_traceback(tb)

File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 903, in call self.fn() if output_subset is None else\

File "pygpu/gpuarray.pyx", line 693, in pygpu.gpuarray.pygpu_empty

File "pygpu/gpuarray.pyx", line 301, in pygpu.gpuarray.array_empty

GpuArrayException: b'cuMemAlloc: CUDA_ERROR_OUT_OF_MEMORY: out of memory' Apply node that caused the error: GpuElemwise{mul,no_inplace}(GpuElemwise{Composite{(exp(i0) / i1)}}[].0, for{gpu,scan_fn}.0) Toposort index: 182 Inputs types: [GpuArrayType(float32, (True, True, False, False, True, True)), GpuArrayType(float32, (False, False, False, True, False, False))] Inputs shapes: [(1, 1, 41, 41, 1, 1), (2200, 41, 41, 1, 41, 4)] Inputs strides: [(6724, 6724, 164, 4, 4, 4), (1102736, 26896, 656, 656, 16, 4)] Inputs values: ['not shown', 'not shown'] Outputs clients: [[Shape(GpuElemwise{mul,no_inplace}.0), GpuReshape{3}(GpuElemwise{mul,no_inplace}.0, MakeVector{dtype='int64'}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer): File "/home/raghav_menon/ZeroResource/SABC/Data_manip_Caps_net.py", line 153, in caps_nn = Caps_nn(X,y_train_int,frame_length,feat_ndims,Epoch) # Training Input without normalisation File "/home/raghav_menon/ZeroResource/SABC/Capsnet.py", line 224, in Caps_nn caps_nn = Caps_NN(Epoches,frame_length,feat_ndims,X_train.shape[0],y_train).fit(X_train, y_train) File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/nolearn/lasagne/base.py", line 690, in fit self.initialize() File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/nolearn/lasagne/base.py", line 490, in initialize self.y_tensor_type, File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/nolearn/lasagne/base.py", line 621, in _create_iter_funcs layers, target=y_batch, objective_kw) File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/nolearn/lasagne/base.py", line 193, in objective output_layer, deterministic=deterministic, get_output_kw) File "/home/raghav_menon/anaconda3/lib/python3.6/site-packages/lasagne/layers/helper.py", line 190, in get_output all_outputs[layer] = layer.get_output_for(layer_inputs, *kwargs) File "/home/raghav_menon/ZeroResource/SABC/CapsLayer.py", line 85, in get_output_for outputs = squash(T.sum(c inputs_hat, 1, keepdims=True))

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

Thanks.

Regards, Raghav