GpuCrossentropySoftmaxArgmax1HotWithBias in the DNN example block 38

[gforge]

Everything runs fine when running on the CPU while when I run the same code on my Tesla K40c device I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-38-45ae0203312f> in <module>()
      6 for epoch in xrange(n_epochs):
      7     for minibatch_index in xrange(n_train_batches):
----> 8         minibatch_avg_cost = train_model(minibatch_index)
      9 
     10     validation_losses = [validate_model(i) for i

/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    586                     # For the CVM
    587                     gof.vm.raise_with_op(self.fn.nodes[self.fn.position_of_error],
--> 588                                          self.fn.thunks[self.fn.position_of_error])
    589                 else:
    590                     # For the c linker

/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.pyc in __call__(self, *args, **kwargs)
    577         t0_fn = time.time()
    578         try:
--> 579             outputs = self.fn()
    580         except Exception:
    581             if hasattr(self.fn, 'position_of_error'):

RuntimeError: Cuda error: GpuCrossentropySoftmaxArgmax1HotWithBias node_6b7275e07874be7aaf65352648b8c2d7_0: invalid configuration argument.
The kernel was launched with 1 threads, 0 blocks and 0 shared memory

Apply node that caused the error: GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0, b, GpuElemwise{Identity{output_types_preference=<theano.scalar.basic.specific_out object at 0x7f2984c598d0>},no_inplace}.0)
Inputs shapes: [(0, 10), (10,), (0,)]
Inputs strides: [(10, 1), (1,), (1,)]
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, vector), CudaNdarrayType(float32, vector)]
Use the Theano flag 'exception_verbosity=high' for a debugprint of this apply node.

I run on a Ubuntu 14.04 system with a basic .theanorc config:

~$ cat .theanorc 
[global]
floatX = float32
device = gpu0

[nvcc]
fastmath = True

Any clue to what may be causing this error?

Thanks by the way for an excellent tutorial!

[graphific]

Hey gforge, which version of theano are you using? Google helps me find: https://groups.google.com/forum/#!topic/theano-users/IHKB_vxKJrU

can you try running it with theano flag "exception_verbosity=high" ?

[gforge]

It is the 0.7.0 version together with CUDA 7.0. A slightly surprising detail is that Alec Radford's almost identical tutorials work (https://github.com/Newmu/Theano-Tutorials). Once I'm back at work I'll try with the verbose command.

[gforge]

Sorry for the delay, here's the error with verbosity high:

RuntimeError: Cuda error: GpuCrossentropySoftmaxArgmax1HotWithBias node_6b7275e07874be7aaf65352648b8c2d7_0: invalid configuration argument.
The kernel was launched with 1 threads, 0 blocks and 0 shared memory

Apply node that caused the error: GpuCrossentropySoftmaxArgmax1HotWithBias(GpuDot22.0, b, GpuElemwise{Identity{output_types_preference=<theano.scalar.basic.specific_out object at 0x7f254fd261d0>},no_inplace}.0)
Inputs shapes: [(0, 10), (10,), (0,)]
Inputs strides: [(10, 1), (1,), (1,)]
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, vector), CudaNdarrayType(float32, vector)]
Debugprint of the apply node: 
GpuCrossentropySoftmaxArgmax1HotWithBias.0 [@A] <CudaNdarrayType(float32, vector)> ''   
 |GpuDot22 [@B] <CudaNdarrayType(float32, matrix)> ''   
 | |GpuElemwise{Composite{[tanh(add(i0, i1))]}}[(0, 0)] [@C] <CudaNdarrayType(float32, matrix)> ''   
 | | |GpuDot22 [@D] <CudaNdarrayType(float32, matrix)> ''   
 | | | |GpuSubtensor{int32:int32:} [@E] <CudaNdarrayType(float32, matrix)> ''   
 | | | | |<CudaNdarrayType(float32, matrix)> [@F] <CudaNdarrayType(float32, matrix)>
 | | | | |ScalarFromTensor [@G] <int32> ''   
 | | | | | |Elemwise{mul,no_inplace} [@H] <TensorType(int32, scalar)> ''   
 | | | | |   |TensorConstant{200} [@I] <TensorType(int32, scalar)>
 | | | | |   |<TensorType(int32, scalar)> [@J] <TensorType(int32, scalar)>
 | | | | |ScalarFromTensor [@K] <int32> ''   
 | | | |   |Elemwise{Composite{[mul(i0, add(i1, i2))]}} [@L] <TensorType(int32, scalar)> ''   
 | | | |     |TensorConstant{200} [@I] <TensorType(int32, scalar)>
 | | | |     |TensorConstant{1} [@M] <TensorType(int32, scalar)>
 | | | |     |<TensorType(int32, scalar)> [@J] <TensorType(int32, scalar)>
 | | | |W [@N] <CudaNdarrayType(float32, matrix)>
 | | |GpuDimShuffle{x,0} [@O] <CudaNdarrayType(float32, row)> ''   
 | |   |b [@P] <CudaNdarrayType(float32, vector)>
 | |W [@Q] <CudaNdarrayType(float32, matrix)>
 |b [@R] <CudaNdarrayType(float32, vector)>
 |GpuElemwise{Identity{output_types_preference=<theano.scalar.basic.specific_out object at 0x7f254fd261d0>},no_inplace} [@S] <CudaNdarrayType(float32, vector)> ''   
   |GpuSubtensor{int32:int32:} [@T] <CudaNdarrayType(float32, vector)> ''   
     |<CudaNdarrayType(float32, vector)> [@U] <CudaNdarrayType(float32, vector)>
     |ScalarFromTensor [@G] <int32> ''   
     |ScalarFromTensor [@K] <int32> ''   
GpuCrossentropySoftmaxArgmax1HotWithBias.1 [@A] <CudaNdarrayType(float32, matrix)> ''   
GpuCrossentropySoftmaxArgmax1HotWithBias.2 [@A] <CudaNdarrayType(float32, vector)> ''   

[graphific]

looks to me like a float32 <-> int32 mismatch, might just be an error in the code I hacked together for the meetup. More stable working code indeed by Alec Radford, or otherwise at deeplearning.net/tutorial/