Calling Vgg16() in lesson1.ipynb gives error: The shape of the input to "Flatten" is not fully defined (got (0, 7, 512)

[bluelight773]

Following installation of all requirements for Part 1 on Ubuntu 16 / Python 3.5:

If I just go through lesson1.ipynb (part 1), which has the dogscats path, as soon as I try to execute vgg=Vgg16() I get the error below:

ValueError Traceback (most recent call last)

in () ----> 1 vgg = Vgg16() /home/bluelight/projects/fastai1/deeplearning_keras2/nbs/vgg16.py in __init__(self) 30 def __init__(self): 31 self.FILE_PATH = 'http://files.fast.ai/models/' ---> 32 self.create() 33 self.get_classes() 34 /home/bluelight/projects/fastai1/deeplearning_keras2/nbs/vgg16.py in create(self) 73 self.ConvBlock(3, 512) 74 ---> 75 model.add(Flatten()) 76 self.FCBlock() 77 self.FCBlock() /home/bluelight/Envs/fastai1py3k2/lib/python3.5/site-packages/keras/models.py in add(self, layer) 461 output_shapes=[self.outputs[0]._keras_shape]) 462 else: --> 463 output_tensor = layer(self.outputs[0]) 464 if isinstance(output_tensor, list): 465 raise TypeError('All layers in a Sequential model ' /home/bluelight/Envs/fastai1py3k2/lib/python3.5/site-packages/keras/engine/topology.py in __call__(self, inputs, **kwargs) 581 # Infering the output shape is only relevant for Theano. 582 if all([s is not None for s in _to_list(input_shape)]): --> 583 output_shape = self.compute_output_shape(input_shape) 584 else: 585 if isinstance(input_shape, list): /home/bluelight/Envs/fastai1py3k2/lib/python3.5/site-packages/keras/layers/core.py in compute_output_shape(self, input_shape) 486 raise ValueError('The shape of the input to "Flatten" ' 487 'is not fully defined ' --> 488 '(got ' + str(input_shape[1:]) + '. ' 489 'Make sure to pass a complete "input_shape" ' 490 'or "batch_input_shape" argument to the first ' ValueError: The shape of the input to "Flatten" is not fully defined (got (0, 7, 512). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.

[roebius]

I have updated the _keras.json.forTheano template in the repo to the latest Keras format. If you download the new file, rename it as "keras.json" and put it in the .keras folder in place of the old one the issue should hopefully disappear.

[vteodorescu]

I had the same error.

I added this to the import section of vgg16.py:

from keras import backend as K K.set_image_dim_ordering('th')

fixed the problem :)

[vteodorescu]

There is also this pull request in the main fastai repo:

https://github.com/fastai/courses/commit/5efc3c79d5b0b9f0cd21373b0ef17612864ea721

[roebius]

I updated the keras.json for Theano anyway, since the previous version in my repo was obsolete. Many thanks @vteodorescu for adding your suggestion.

[vteodorescu]

I spoke too early - I passed over the shape issue, now I get an 'out of memory' error...

I have 4GB of GPU memory and 8GB of RAM memory

Back to do more research, I will try and use vgg16 straight from the keras repo, see if it helps

[roebius]

I assume you already tested reducing the batch size, otherwise it would be worth a try.

On 27 Sep 2017, at 19:13, Viorel Emilian Teodorescu notifications@github.com wrote:

I spoke too early - I passed over the shape issue, now I get an 'out of memory' error...

I have 4GB of GPU memory and 8GB of RAM memory

Back to do more research, I will try and use vgg16 straight from the keras repo, see if it helps

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

[vteodorescu]

"slaps forehead v hard... :) "...

I did not try it before you mentioned, thank you for the tip, roebius! I was getting version fatigue a bit.

Dropped to batches = 16 and it did run finally. With batches 32 it ran out of memory on the last iteration... :) :) :)

On to the next challenge!

All the best!

[hamedkhanpour]

I ran lesson 1 on AWS p2.xlarge instance and the following error pops up:

ImportError Traceback (most recent call last)

in () 1 # check that ~/.keras/keras.json is set for Theano and includes "image_data_format": "channels_first" 2 from imp import reload # Python 3 ----> 3 import utils; reload(utils) 4 from utils import plots /home/ubuntu/fastai2/nbs/utils.py in () 41 from keras.models import Sequential, Model 42 from keras.layers import Input, Embedding, Reshape, merge, LSTM, Bidirectional ---> 43 from keras.layers import SpatialDropout1D, Concatenate # Keras2 44 45 from keras.layers import TimeDistributed, Activation, SimpleRNN, GRU ImportError: cannot import name SpatialDropout1D It seems that it complains about SpatialDropout1D....how can I solve this? Thanks much!!

[roebius]

SpatialDropout1D was introduced with Keras 2. Could you check which version of Keras you have installed?

In order to get the version launch Python in a terminal session and type the following: import keras print(keras.__version__)

If you have the right version of Keras then the following should work too: from keras.layers import SpatialDropout1D

[hamedkhanpour]

Thanks for the quick response! I update Keras to 2.08 on AWS and I get the following error now:

---

ValueError Traceback (most recent call last)

in () ----> 1 vgg = Vgg16() 2 # Grab a few images at a time for training and validation. 3 # NB: They must be in subdirectories named based on their category 4 batches = vgg.get_batches(path+'train', batch_size=batch_size) 5 val_batches = vgg.get_batches(path+'valid', batch_size=batch_size*2) /home/ubuntu/fastai2/nbs/vgg16.pyc in __init__(self) 30 def __init__(self): 31 self.FILE_PATH = 'http://files.fast.ai/models/' ---> 32 self.create() 33 self.get_classes() 34 /home/ubuntu/fastai2/nbs/vgg16.pyc in create(self) 73 self.ConvBlock(3, 512) 74 ---> 75 model.add(Flatten()) 76 self.FCBlock() 77 self.FCBlock() /home/ubuntu/anaconda2/lib/python2.7/site-packages/keras/models.pyc in add(self, layer) 473 output_shapes=[self.outputs[0]._keras_shape]) 474 else: --> 475 output_tensor = layer(self.outputs[0]) 476 if isinstance(output_tensor, list): 477 raise TypeError('All layers in a Sequential model ' /home/ubuntu/anaconda2/lib/python2.7/site-packages/keras/engine/topology.pyc in __call__(self, inputs, **kwargs) 619 # Infering the output shape is only relevant for Theano. 620 if all([s is not None for s in _to_list(input_shape)]): --> 621 output_shape = self.compute_output_shape(input_shape) 622 else: 623 if isinstance(input_shape, list): /home/ubuntu/anaconda2/lib/python2.7/site-packages/keras/layers/core.pyc in compute_output_shape(self, input_shape) 475 raise ValueError('The shape of the input to "Flatten" ' 476 'is not fully defined ' --> 477 '(got ' + str(input_shape[1:]) + '. ' 478 'Make sure to pass a complete "input_shape" ' 479 'or "batch_input_shape" argument to the first ' ValueError: The shape of the input to "Flatten" is not fully defined (got (0, 7, 512). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.

[roebius]

Check your keras.json file: it looks like in your case it contains settings for TensorFlow instead of Theano. Look also at my first comment in this thread.

[roebius]

@hamedkhanpour Not related to the last error that you experienced above: I have noted from your ValueError Traceback that you are using Python 2.7. Please keep in mind that the notebooks in my repo have been tested with Python 3.5 on Ubuntu 16.04 (part 1 and 2) and Python 3.6 on Mac OSX 10.12.5 (part 1 only). I have no plans involving the use of Python 2.