CPU error

[jonathanort]

I changed line 16 in generate.py to from lasagne.layers import Conv2DLayer as ConvLayer

But then this error happens:

storyteller/romance.npz Loading skip-thoughts... /usr/local/lib/python2.7/site-packages/theano/scan_module/scan.py:1017: Warning: In the strict mode, all neccessary shared variables must be passed as a part of non_sequences 'must be passed as a part of non_sequences', Warning) Loading decoder... Loading image-sentence embedding... Loading and initializing ConvNet... Loading parameters... Loading captions... Embedding captions... Loading biases... Traceback (most recent call last): File "index.py", line 3, in generate.story(z, './images/ex1.jpg') File "/Users/jonathan/Desktop/Story/generate.py", line 41, in story feats = compute_features(z['net'], im).flatten() File "/Users/jonathan/Desktop/Story/generate.py", line 164, in compute_features fc7 = numpy.array(lasagne.layers.get_output(net['fc7'], im, deterministic=True).eval()) File "/usr/local/lib/python2.7/site-packages/lasagne/layers/helper.py", line 185, in get_output all_outputs[layer] = layer.get_output_for(layer_inputs, **kwargs) File "/usr/local/lib/python2.7/site-packages/lasagne/layers/pool.py", line 240, in get_output_for mode=self.mode, TypeError: max_pool_2d() got an unexpected keyword argument 'mode'

Is there any good solution?

[YknZhu]

Can you try this fork?

This is a temporary solution that uses Caffe for convnet. You need to change the flag in line 8 of config.py to True for running on CPU. I personally suggest you to grab a coffee while running generate.load_all() as it is super slow in CPU.

[breezedeus]

I got the same error as jonathanort did on CPU... And, it's super slow, generate.load_all() costs me almost two days.

[YknZhu]

Did you update to the latest commit? If you have set FLAG_CPU_MODE = True then fc7 = numpy.array(lasagne.layers.get_output(net['fc7'], im, deterministic=True).eval()) won't be executed.

generate.load_all() should't take two days... Most time is spent on embedding captions, and it generally takes 10mins or so on a workstation with 32GB memory.

[breezedeus]

I am trying with your new fork, but still very slow. I do that on a VM, maybe that's why slow. Can multi-CPU help? Maybe I should try on a physical PC.

Right now, it stops here:

[YknZhu]

Running on VM is not a good idea. Skipthoughts takes large amount of memory while CNN forwarding / embedding captions require considerable computation power.

Here attaches the printout message on my workstation. It takes ~2mins to get here and 8mins more for embedding captions.

...
I1116 22:54:35.138017  1391 net.cpp:283] This network produces output prob
I1116 22:54:35.138062  1391 net.cpp:297] Network initialization done.
I1116 22:54:35.138077  1391 net.cpp:298] Memory required for data: 1250445440
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message.  If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 574671192
I1116 22:54:39.202268  1391 upgrade_proto.cpp:618] Attempting to upgrade input file specified using deprecated V1LayerParameter: /ais/guppy9/movie2text/neural-storyteller/models/VGG_ILSVRC_19_layers.caffemodel
I1116 22:54:40.283370  1391 upgrade_proto.cpp:626] Successfully upgraded file specified using deprecated V1LayerParameter
Loading captions...
Embedding captions...

[breezedeus]

@YknZhu Thanks. I'll try on a real machine, and tell the result later.

[breezedeus]

I can get stories right now, thx. Most of the generated stories seem confusing, hard to understand what about...

[YknZhu]

Can you try generate.story(z, './images/ex1.jpg', k=100, bw=50) to confirm you get the same story as in README.md?

[breezedeus]

When I change to swift_style.npy, the output is "Hey" for generate.story(z, './images/ex1.jpg', k=100, bw=50, lyric=True) ...

NEAREST-CAPTIONS: A surfer is silhouetted against a sunset at the beach . a person rides a horse at sunset along the beach A person rides a horse on a beach at sunset . A surfer looks at the ocean while the sun is setting . people look out at the ocean as some birds fly over

OUTPUT: Hey

[breezedeus]

@YknZhu Any suggestion?

[YknZhu]

Sorry for my late reply. Getting super busy these days.

Do you get the same result as shown in README.md by setting lyric=False (and use romance_style.npy)?

[breezedeus]

@YknZhu Yes, I can get the same result.

[YknZhu]

That is weird. I cannot reproduce your result on my end. Could you try adding

[global]
floatX = float32

in ~/.theanorc

or try running it on GPU?