search

keras-team / keras

Deep Learning for humans
http://keras.io/
Apache License 2.0
61.9k stars 19.45k forks source link

After updating, return errors! #2046

Closed Imorton-zd closed 8 years ago

Imorton-zd commented 8 years ago 

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 699, in runfile
    execfile(filename, namespace)
  File "C:\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)
  File "E:/DL EX/project/examples/keras-master/keras-master/examples/imdb_lstm.py", line 57, in <module>
    optimizer='adam')
  File "C:\Anaconda2\lib\site-packages\keras\models.py", line 564, in compile
    updates=updates, **kwargs)
  File "C:\Anaconda2\lib\site-packages\keras\backend\theano_backend.py", line 460, in function
    return Function(inputs, outputs, updates=updates, **kwargs)
  File "C:\Anaconda2\lib\site-packages\keras\backend\theano_backend.py", line 446, in __init__
    allow_input_downcast=True, **kwargs)
  File "C:\Anaconda2\lib\site-packages\theano\compile\function.py", line 320, in function
    output_keys=output_keys)
  File "C:\Anaconda2\lib\site-packages\theano\compile\pfunc.py", line 479, in pfunc
    output_keys=output_keys)
  File "C:\Anaconda2\lib\site-packages\theano\compile\function_module.py", line 1779, in orig_function
    defaults)
  File "C:\Anaconda2\lib\site-packages\theano\compile\function_module.py", line 1643, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "C:\Anaconda2\lib\site-packages\theano\gof\link.py", line 690, in make_thunk
    storage_map=storage_map)[:3]
  File "C:\Anaconda2\lib\site-packages\theano\gof\vm.py", line 1003, in make_all
    no_recycling))
  File "C:\Anaconda2\lib\site-packages\theano\sandbox\cuda\__init__.py", line 256, in make_thunk
    compute_map, no_recycling)
  File "C:\Anaconda2\lib\site-packages\theano\gof\op.py", line 970, in make_thunk
    no_recycling)
  File "C:\Anaconda2\lib\site-packages\theano\gof\op.py", line 879, in make_c_thunk
    output_storage=node_output_storage)
  File "C:\Anaconda2\lib\site-packages\theano\gof\cc.py", line 1200, in make_thunk
    keep_lock=keep_lock)
  File "C:\Anaconda2\lib\site-packages\theano\gof\cc.py", line 1143, in __compile__
    keep_lock=keep_lock)
  File "C:\Anaconda2\lib\site-packages\theano\gof\cc.py", line 1595, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "C:\Anaconda2\lib\site-packages\theano\gof\cmodule.py", line 1142, in module_from_key
    module = lnk.compile_cmodule(location)
  File "C:\Anaconda2\lib\site-packages\theano\gof\cc.py", line 1506, in compile_cmodule
    preargs=preargs)
  File "C:\Anaconda2\lib\site-packages\theano\sandbox\cuda\nvcc_compiler.py", line 403, in compile_str
    'for cmd', ' '.join(cmd))

What happened?

Imorton-zd commented 8 years ago

I have used git+git to update the newest theano and keras. Please give me some suggestions. Thanks a lot.

NasenSpray commented 8 years ago

When did you update Theano? It was broken on Windows up until 2 days ago.

Imorton-zd commented 8 years ago

@NasenSpray Today. Can I change the Theano for an old version?

NasenSpray commented 8 years ago

Oh wait, it's probably something different. Are the other examples working (e.g. MNIST CNN)? And when did you update Keras the last time before today?

NasenSpray commented 8 years ago

@nouiz: that's something for you, a code-gen bug on Windows mod.cu(9701) : fatal error C1061: compiler limit : blocks nested too deeply

Exception: ('The following error happened while compiling the node', GpuJoin(Ten sorConstant{1}, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0)

Whatever monstrosity Theano is trying to compile, it exceeds MSVC's capabilities. I already observed this problem a while ago and it seems to be caused by some weird shape-dependent interaction involving scan&dot (e.g.in imdb_lstm.py, it only happens when the LSTM has 120 nodes or more)

@Imorton-zd

You can revert Keras to 06a1545645d974350d13425246eec53a08cb6ab8, the RNN implementations in there don't trigger the bug.

nouiz commented 8 years ago

Can you some give me the code that reproduce this?

This is caused by GpuJoin. We unroll a loop in C, on the number of inputs of the node. It seem you hit a case where there is too many inputs for MSVC. I can easily change that to a for loop in C with some working example.

Fred

On Wed, Mar 23, 2016 at 7:04 AM, NasenSpray notifications@github.com wrote:

@nouiz https://github.com/nouiz: that's something for you, a code-gen bug on Windows

mod.cu(9701) : fatal error C1061: compiler limit : blocks nested too deeply

Exception: ('The following error happened while compiling the node', GpuJoin(Ten sorConstant{1}, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc .0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuAlloc.0, GpuA lloc.0, GpuAlloc.0, GpuAlloc.0)

Whatever monstrosity Theano is trying to compile, it exceeds MSVC's capabilities. I already observed this problem a while ago and it seems to be caused by some weird shape-dependent interaction involving scan&dot ( e.g.in imdb_lstm.py https://github.com/fchollet/keras/blob/master/examples/imdb_lstm.py#L50, it only happens when the LSTM has 120 nodes or more)

@Imorton-zd https://github.com/Imorton-zd

You can revert Keras to 06a1545 https://github.com/fchollet/keras/commit/06a1545645d974350d13425246eec53a08cb6ab8, the RNN implementations in there don't trigger the bug.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/fchollet/keras/issues/2046#issuecomment-200297795

NasenSpray commented 8 years ago

@nouiz: imdb_lstm.py, or do you need one without Keras?

NasenSpray commented 8 years ago

@nouiz: I made a fix, relevant stuff here. Please take a look, I've used some C++11 stuff and wanted to ask whether that's ok before I submit a PR.

nouiz commented 8 years ago

thanks for working on this.

I don't like adding as a dependency a new version of C++, as I'm pretty sure there is still people that use OS that don't have it. So they won't be able to update Theano without updating there OS. Which isn't great. Don't forget, C++11 is when the standard was accepted. It takes some times to be implemented by compiler and then only after, it get added to new version of OS.

If someone find when Windows, Mac, Ubuntu and CentOS started to have it available, we can then start to talk if we can do it now. Without that information, we risk too much to prevent some user to update. But sure, it make the code much more readable and less error prone. It would be great that we can do it.

Don't hesitate to make a PR, it make it easier to talk about code.

On Wed, Mar 23, 2016 at 2:25 PM, NasenSpray notifications@github.com wrote:

@nouiz https://github.com/nouiz: I made a fix, relevant stuff here https://gist.github.com/NasenSpray/357f7767e457935bbef9. Please take a look, I've used some C++11 stuff and wanted to ask whether that's ok before I submit a PR.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/fchollet/keras/issues/2046#issuecomment-200480789

NasenSpray commented 8 years ago

C++11 is supported on basically every platform since ~3 years. GCC actually defaults to C++14 nowadays :D

Don't hesitate to make a PR, it make it easier to talk about code.

oke, I'm on it

[edit] https://github.com/Theano/Theano/pull/4289