couldn't run on gpu

[iamsile]

Hi, @saltypaul , after update, I run it on tesla k20c, but when on the 300/20000 epoch, there is something wrong in the program.

Traceback (most recent call last): File "build_model.py", line 76, in model = build_model(retrain=True) File "build_model.py", line 55, in build_model loss, costs = model.train(enIpt, enMsk, deIpt, deMsk, deTgt) File "/export/taowei/work/Seq2Seq/src/seq2seq.py", line 175, in train return self._train(encoderInputs, encoderMask, decoderInputs, decoderMask, decoderTarget) File "/usr/lib/python2.7/site-packages/theano/compile/function_module.py", line 875, in call storage_map=getattr(self.fn, 'storage_map', None)) File "/usr/lib/python2.7/site-packages/theano/gof/link.py", line 325, in raise_with_op reraise(exc_type, exc_value, exc_trace) File "/usr/lib/python2.7/site-packages/theano/compile/function_module.py", line 862, in call self.fn() if output_subset is None else\ IndexError: One of the index value is out of bound. Error code: 65535.\n Apply node that caused the error: GpuAdvancedSubtensor1(Encoder LookUpTable, Elemwise{Cast{int64}}.0) Toposort index: 98 Inputs types: [CudaNdarrayType(float32, matrix), TensorType(int64, vector)] Inputs shapes: [(29331, 512), (500,)] Inputs strides: [(512, 1), (8,)] Inputs values: ['not shown', 'not shown'] Outputs clients: [[GpuReshape{3}(GpuAdvancedSubtensor1.0, MakeVector{dtype='int64'}.0)]]

Do you know how to fix it? Thank you very much!

[damientseng]

What data set are you using? The "Encoder LookUpTable" has shape (vocabulary_size, hidden_size). The build_model method accepts as the first argument the vocabulary_size, which should by the size of the token set of you data.

[iamsile]

I use the origin data in the project, and I didn't modify any parameters. I set the shape is (vocabulary_size, hidden_size)

[damientseng]

That's odd...my vocabulary size is 29331, because in make_convs.py I trimed off tokens that appear less than 5 time in total. Obvious your real vocabulary size is much larger.

[iamsile]

Thank you ! Let me try to solve it.

[iamsile]

@saltypaul , you're right! It's my fault which I set a small vocab_size ! Thank you very much to help me ! Thank you!