Closed carlthome closed 7 years ago
Make sure to use Theano dev version. If you where using it, update it again. I fixed a crash that could solve this problem recently.
If the problem persist, give the full error message.
On Sun, Nov 6, 2016 at 2:15 PM, Carl Thomé notifications@github.com wrote:
KeyError: ('The following error happened while compiling the node', forall_inplace,cpu,scan_fn}(Shape_i{1}.0, InplaceGpuDimShuffle{0,1,4,2,3}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, GpuIncSubtensor{InplaceSet;:int64:}.0, Shape_i{1}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{x,x,x,0}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{x,x,x,0}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{x,x,x,0}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{3,2,0,1}.0, InplaceGpuDimShuffle{x,x,x,0}.0, InplaceGpuDimShuffle{3,2,0,1}.0), '\n', 'The following error happened while compiling the node', GpuDnnConv{algo='small', inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0, GpuDnnConvDesc{border_mode='half', subsample=(1, 1), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0}), '\n', 'cudnn_handle')
I assume this is related to having convolution operations inside a scan. @jeammimi https://github.com/jeammimi, @nouiz https://github.com/nouiz, @fchollet https://github.com/fchollet, what do you think? Can you get the conv_lstm.py example working with THEANO_FLAGS=device=cuda and cuDNN?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fchollet/keras/issues/4305, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC-xFwY1tG7Beq2Fm_A_NYaZGJpbinks5q7ic4gaJpZM4KqpbU .
Problem persists with Theano's latest unfortunately (installed from GitHub master just now). Please try python keras/examples/conv_lstm.py
, same for you?
pip show theano
:
Name: Theano
Version: 0.9.0.dev4
Summary: Optimizing compiler for evaluating mathematical expressions on CPUs and GPUs.
Home-page: http://deeplearning.net/software/theano/
Author: LISA laboratory, University of Montreal
Author-email: theano-dev@googlegroups.com
License: BSD
Location: /home/carl/anaconda3/lib/python3.5/site-packages
Requires: numpy, scipy, six
python conv_lstm.py
:
Using Theano backend.
Mapped name None to device cuda: TITAN X (Pascal)
Using cuDNN version 5105 on context None
Traceback (most recent call last):
File "conv_lstm.py", line 104, in <module>
nb_epoch=300, validation_split=0.05)
File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/models.py", line 640, in fit
sample_weight=sample_weight)
File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/engine/training.py", line 1084, in fit
self._make_test_function()
File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/engine/training.py", line 739, in _make_test_function
**self._function_kwargs)
File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 821, in function
return Function(inputs, outputs, updates=updates, **kwargs)
File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 807, in __init__
**kwargs)
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/compile/function.py", line 326, in function
output_keys=output_keys)
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/compile/pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py", line 1784, in orig_function
defaults)
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py", line 1651, in create
input_storage=input_storage_lists, storage_map=storage_map)
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/link.py", line 699, in make_thunk
storage_map=storage_map)[:3]
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/vm.py", line 1057, in make_all
impl=impl))
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/op.py", line 924, in make_thunk
no_recycling)
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/op.py", line 824, in make_c_thunk
no_recycling=e_no_recycling)
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/cc.py", line 563, in accept
self.fetch_variables()
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/cc.py", line 589, in fetch_variables
params = node.run_params()
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/graph.py", line 129, in run_params
return self.op.get_params(self)
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gpuarray/dnn.py", line 215, in get_params
ptr = get_prop(self.dnn_context(node), 'cudnn_handle').value
File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gpuarray/type.py", line 114, in get_prop
return _get_props(name)[k]
KeyError: ('The following error happened while compiling the node', GpuDnnConv{algo='small', inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0, GpuDnnConvDesc{border_mode='half', subsample=(1, 1), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0}), '\n', 'cudnn_handle')
What Theano and Keras flag do you use?
On Mon, Nov 7, 2016 at 10:05 AM, Carl Thomé notifications@github.com wrote:
Problem persists with Theano's latest unfortunately (installed from GitHub master just now). Please try python keras/examples/conv_lstm.py, same for you?
pip show theano:
Name: Theano Version: 0.9.0.dev4 Summary: Optimizing compiler for evaluating mathematical expressions on CPUs and GPUs. Home-page: http://deeplearning.net/software/theano/ Author: LISA laboratory, University of Montreal Author-email: theano-dev@googlegroups.com License: BSD Location: /home/carl/anaconda3/lib/python3.5/site-packages Requires: numpy, scipy, six
python conv_lstm.py:
Using Theano backend. Mapped name None to device cuda: TITAN X (Pascal) Using cuDNN version 5105 on context None Traceback (most recent call last): File "conv_lstm.py", line 104, in
nb_epoch=300, validation_split=0.05) File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/models.py", line 640, in fit sample_weight=sample_weight) File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/engine/training.py", line 1084, in fit self._make_test_function() File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/engine/training.py", line 739, in _make_test_function _self._function_kwargs) File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 821, in function return Function(inputs, outputs, updates=updates, _kwargs) File "/home/carl/anaconda3/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 807, in init **kwargs) File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/compile/function.py", line 326, in function output_keys=output_keys) File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/compile/pfunc.py", line 486, in pfunc output_keys=output_keys) File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py", line 1784, in orig_function defaults) File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/compile/function_module.py", line 1651, in create input_storage=input_storage_lists, storage_map=storage_map) File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/link.py", line 699, in make_thunk storage_map=storage_map)[:3] File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/vm.py", line 1057, in make_all impl=impl)) File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/op.py", line 924, in make_thunk no_recycling) File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/op.py", line 824, in make_c_thunk no_recycling=e_no_recycling) File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/cc.py", line 563, in accept self.fetch_variables() File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/cc.py", line 589, in fetch_variables params = node.run_params() File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gof/graph.py", line 129, in run_params return self.op.get_params(self) File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gpuarray/dnn.py", line 215, in get_params ptr = get_prop(self.dnn_context(node), 'cudnn_handle').value File "/home/carl/anaconda3/lib/python3.5/site-packages/theano/gpuarray/type.py", line 114, in get_prop return _get_props(name)[k] KeyError: ('The following error happened while compiling the node', GpuDnnConv{algo='small', inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0, GpuDnnConvDesc{border_mode='half', subsample=(1, 1), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0}), '\n', 'cudnn_handle') — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fchollet/keras/issues/4305#issuecomment-258859712, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC-7K2gEtXJqeJ9ROvSrupdVC8K1Rhks5q7z5HgaJpZM4KqpbU .
Pretty standard:
[global]
device = cuda
floatX = float32
{
"floatx": "float32",
"backend": "theano",
"image_dim_ordering": "th",
"epsilon": 1e-07
}
Same error with image_dim_ordering='tf'
as well.
Annoyingly, downgrading (e.g. pip install git+git://github.com/theano/theano@1dabf8540db2f5eda4a80c2798848c12836642c1
) is the best fix I could come up with at the moment. :cry:
I've tried to reproduce this, but I get a different error:
$ THEANO_FLAGS=floatX=float32,device=cuda0 python conv_lstm.py
Using Theano backend.
Mapped name None to device cuda0: GeForce GTX 750 Ti
PCI Bus ID: 0000:07:00.0
Using cuDNN version 5105 on context None
/home/anakha/ext/keras/keras/layers/convolutional_recurrent.py:279: UserWarning: Be carefull if used with convolution3D layers:
th in convolution 3D corresponds to (samples, channels, conv_dim1, conv_dim2,conv_dim3)
while for this network it corresponds to: (samples, time, channels, rows, cols)
warnings.warn('Be carefull if used with convolution3D layers:\n'
/home/anakha/ext/keras/keras/layers/convolutional_recurrent.py:279: UserWarning: Be carefull if used with convolution3D layers:
th in convolution 3D corresponds to (samples, channels, conv_dim1, conv_dim2,conv_dim3)
while for this network it corresponds to: (samples, time, channels, rows, cols)
warnings.warn('Be carefull if used with convolution3D layers:\n'
Traceback (most recent call last):
File "conv_lstm.py", line 104, in <module>
nb_epoch=300, validation_split=0.05)
File "/home/anakha/ext/keras/keras/models.py", line 642, in fit
sample_weight=sample_weight)
File "/home/anakha/ext/keras/keras/engine/training.py", line 1062, in fit
batch_size=batch_size)
File "/home/anakha/ext/keras/keras/engine/training.py", line 1000, in _standardize_user_data
check_loss_and_target_compatibility(y, self.loss_functions, self.internal_output_shapes)
File "/home/anakha/ext/keras/keras/engine/training.py", line 215, in check_loss_and_target_compatibility
' while using as loss `' + loss.__name__ + '`. '
Exception: A target array with shape (1000, 15, 40, 40, 1) was passed for an output of shape (None, None, 40, 40, 1) while using as loss `binary_crossentropy`. This loss expects targets to have the same shape as the output
@kilotaras, https://github.com/fchollet/keras/commit/6b04add93209f557e265bbd04ab34d5491d463f0 probably introduced @abergeron's problem with examples/conv_lstm.py
Shouldn't the shape checking ignore None
? None
is used for allowing variable lengths during shape inference.
@carlthome sorry my bad, I was checking wrong parameter for None. Will prepare PR with fix (and test) in a couple of minutes.
The fix is in #4458
I tried with the current master of Theano, libgpuarray and keras (with the fix in #4458 on top) and I don't get an error.
Curious, I did a reinstall of Theano, Keras and libgpuarray from their respective master branches just now. All tests with python -c "import pygpu; pygpu.test()"
passes but I still get the same error when running conv_lstm.py:
KeyError: ('The following error happened while compiling the node', GpuDnnConv{algo='small', inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0, GpuDnnConvDesc{border_mode='half', subsample=(1, 1), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0}), '\n', 'cudnn_handle')
What is your GPU? Part of the error tell the opposite, that it is old.
What is your GPU and cuda version? Maybe you need a more recent cuda/driver for your GPU.
On Mon, Nov 21, 2016 at 1:52 PM, Carl Thomé notifications@github.com wrote:
Curious, I did a reinstall of Theano, Keras and libgpuarray from their respective master branches just now. All tests with python -c "import pygpu; pygpu.test()" passes but I still get the same error when running conv_lstm.py:
KeyError: ('The following error happened while compiling the node', GpuDnnConv{algo='small', inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0, GpuDnnConvDesc{border_mode='half', subsample=(1, 1), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0}), '\n', 'cudnn_handle')
I think this might be related with that my particular graphics card is very recent. If I force cuDNN with THEANO_FLAGS, warnings and errors pop up:
RuntimeError: You enabled cuDNN, but we aren't able to use it: Can not compile with cuDNN. We got this error:b"nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).\n/usr/bin/ld: /tmp/tmpxft_00007474_00000000-4_try_flags_4_lxnpcb.o: relocation R_X86_64_32 against `.rodata' can not be used when making a shared object; recompile with -fPIC\n/usr/bin/ld: final link failed: Nonrepresentable section on output\ncollect2: error: ld returned 1 exit status\n"
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fchollet/keras/issues/4305#issuecomment-262031229, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC-w3VqTG-YLoIWnY47AAbl-ir19UTks5rAeiFgaJpZM4KqpbU .
Solved!
I'm on Ubuntu 16.04. For CUDA 7.5, if you recall, the gcc version had to be no newer than 4.9, so many of us used update-alternatives
to lock versions. Now, with CUDA 8 nvcc wants a newer gcc so the solution was to configure update-alternatives
properly.
One would also have to clear Theano's compile cache most likely (theano-cache purge
).
I assume this is related to having convolution operations inside a scan. @jeammimi, @nouiz, @fchollet, what do you think? Can you get the
conv_lstm.py
example working withTHEANO_FLAGS=device=cuda
and cuDNN?