Predict function is showing error

[iqbal-chowdhury]

Hi,

Thank you for this interactive demo of VQA. Anyway while running the prediction cell:

`y_output = model_vqa.predict([question_features, image_features])

labelencoder = joblib.load(label_encoder_file_name) for label in reversed(np.argsort(y_output)[0,-5:]): print str(round(y_output[0,label]*100,2)).zfill(5), "% ", labelencoder.inverse_transform(label)`

Then it showing some error just as below:

`ValueError Traceback (most recent call last)

in () ----> 1 y_output = model_vqa.predict([question_features, image_features]) 2 3 # This task here is represented as a classification into a 1000 top answers 4 # this means some of the answers were not part of training and thus would 5 # not show up in the result. /usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/models.pyc in predict(self, x, batch_size, verbose) 454 if self.model is None: 455 raise Exception('The model needs to be compiled before being used.') --> 456 return self.model.predict(x, batch_size=batch_size, verbose=verbose) 457 458 def predict_on_batch(self, x): /usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/engine/training.pyc in predict(self, x, batch_size, verbose) 1117 f = self.predict_function 1118 return self._predict_loop(f, ins, -> 1119 batch_size=batch_size, verbose=verbose) 1120 1121 def train_on_batch(self, x, y, /usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/engine/training.pyc in _predict_loop(self, f, ins, batch_size, verbose) 837 ins_batch = slice_X(ins, batch_ids) 838 --> 839 batch_outs = f(ins_batch) 840 if type(batch_outs) != list: 841 batch_outs = [batch_outs] /usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/backend/theano_backend.pyc in **call**(self, inputs) 505 def **call**(self, inputs): 506 assert type(inputs) in {list, tuple} --> 507 return self.function(*inputs) 508 509 /usr/local/lib/python2.7/dist-packages/theano/compile/function_module.pyc in **call**(self, _args, *_kwargs) 869 node=self.fn.nodes[self.fn.position_of_error], 870 thunk=thunk, --> 871 storage_map=getattr(self.fn, 'storage_map', None)) 872 else: 873 # old-style linkers raise their own exceptions /usr/local/lib/python2.7/dist-packages/theano/gof/link.pyc in raise_with_op(node, thunk, exc_info, storage_map) 312 # extra long error message in that case. 313 pass --> 314 reraise(exc_type, exc_value, exc_trace) 315 316 /usr/local/lib/python2.7/dist-packages/theano/compile/function_module.pyc in **call**(self, _args, *_kwargs) 857 t0_fn = time.time() 858 try: --> 859 outputs = self.fn() 860 except Exception: 861 if hasattr(self.fn, 'position_of_error'): ValueError: total size of new array must be unchanged Apply node that caused the error: Reshape{3}(Elemwise{Add}[(0, 0)].0, TensorConstant{[ -1 30 512]}) Toposort index: 49 Inputs types: [TensorType(float32, matrix), TensorType(int64, vector)] Inputs shapes: [(3, 512), (3,)] Inputs strides: [(2048, 4), (8,)] Inputs values: ['not shown', array([ -1, 30, 512])] Outputs clients: [[Join(TensorConstant{2}, Reshape{3}.0, Reshape{3}.0, Reshape{3}.0, Reshape{3}.0)]] Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer): File "/usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/engine/topology.py", line 341, in create_input_layer self(x) File "/usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/engine/topology.py", line 485, in **call** self.add_inbound_node(inbound_layers, node_indices, tensor_indices) File "/usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/engine/topology.py", line 543, in add_inbound_node Node.create_node(self, inbound_layers, node_indices, tensor_indices) File "/usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/engine/topology.py", line 148, in create_node output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0])) File "/usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/layers/recurrent.py", line 219, in call preprocessed_input = self.preprocess_input(x) File "/usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/layers/recurrent.py", line 729, in preprocess_input input_dim, self.output_dim, timesteps) File "/usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/layers/recurrent.py", line 38, in time_distributed_dense x = K.reshape(x, (-1, timesteps, output_dim)) File "/usr/local/lib/python2.7/dist-packages/Keras-1.0.2-py2.7.egg/keras/backend/theano_backend.py", line 283, in reshape return T.reshape(x, shape) HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.`

[iamaaditya]

It seems the model is not being loaded. Did you download the VGG weights file (see this) ?

[iqbal-chowdhury]

VGG weights file is downloaded and the image features are also extracted. My theano version is 0.8.2. Keras version is 1.0.2. image_features dimenstion = (1, 4096) question_features dimension = (1, 7, 300)

[iamaaditya]

1. First make sure you are running the code of this repo, which was updated to make changes to work with Keras 1+ (do a git pull origin master ) if you need to get the latest.
2. Run the command using following theano flag  THEANO_FLAGS='exception_verbosity=high' python demo.py and then paste the error here. It seems some variable size mis-match but I do not get the same error.

[iqbal-chowdhury]

Now this error appears for

THEANO_FLAGS='exception_verbosity=high' python demo.py -image_file_name test.jpg -question "Is there a man in the picture?":

`Using Theano backend.

Loading image features ... Loading question features ... Traceback (most recent call last): File "demo.py", line 114, in main() File "demo.py", line 95, in main question_features = get_question_features(unicode(args.question, 'utf-8')) File "demo.py", line 72, in get_question_features word_embeddings = spacy.load('en', vectors='en_glove_cc_300_1m_vectors') File "/home/iqbal/.local/lib/python2.7/site-packages/spacy/init.py", line 16, in load vectors_package = get_package_by_name(vectors, via=via) File "/home/iqbal/.local/lib/python2.7/site-packages/spacy/util.py", line 41, in get_package_by_name lang = get_lang_class(name) File "/home/iqbal/.local/lib/python2.7/site-packages/spacy/util.py", line 28, in get_lang_class raise RuntimeError('Language not supported: %s' % lang) RuntimeError: Language not supported: en_glove_cc_300_1m_vectors `

I tried to put spacy.set_lang_class('en_glove_cc_300_1m_vectors', 'vectors') but that didn't work [it worked for the jupyter demo; when the same error was over there]

[iamaaditya]

You might need to update your spacy or change the model, see here https://github.com/iamaaditya/VQA_Demo/issues/6

[anujshah1003]

Try this two commands to get your glove vectors running

sputnik --name spacy --repository-url http://index.spacy.io install en==1.1.0 sputnik --name spacy en_glove_cc_300_1m_vectors

[iqbal-chowdhury]

Thank you :) issue is solved. I added this line in the demo.py after importing spacy.

spacy.set_lang_class('en_glove_cc_300_1m_vectors', None)

[iamaaditya]

Thanks @anujshah1003 and @iqbal-chowdhury Close #7