arielephrat / vid2speech

Code for "Vid2speech: Speech Reconstruction from Silent Video" ICASSP '17
116 stars 35 forks source link

The shape of the input to "Flatten" is not fully defined #1

Open adamhrv opened 7 years ago

adamhrv commented 7 years ago

When running python predict.py --weight_path ../pretrained_weights/s2.hdf5 Theano throws an error about using Flatten() in the build_model:

File "~/vid2speech/code/train.py", line 100, in build_model model.add(Flatten())

ValueError: The shape of the input to "Flatten" is not fully defined (got (0, 4, 128). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.

I'm using Python3.4 on Ubuntu 14.04 with image_dim_ordering set to th and backend set to theano in keras.json

I bluntly tried adding an input shape to the model definition, but this is not correct: model = Sequential(input_shape=(CHANNELS, FRAME_ROWS, FRAME_COLS))

What is the correct way to pass an input shape to the first layer in the model?

NB: when running test scripts it's helpful to increase the size of your swapfile to 64GB+ (depending on RAM)


Full output:

Using Theano backend.
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10).  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce GTX 1080 (CNMeM is disabled, cuDNN 5105)
Loading data...
Done.
~/vid2speech/code/train.py:64: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(32, (3, 3), input_shape=(9, 128, 1..., padding="same", kernel_initializer="he_normal")`
  model.add(Convolution2D(32, 3, 3, border_mode='same', init='he_normal', input_shape=(CHANNELS, FRAME_ROWS, FRAME_COLS)))
~/vid2speech/code/train.py:68: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(32, (3, 3), padding="same", kernel_initializer="he_normal")`
  model.add(Convolution2D(32, 3, 3, border_mode='same', init='he_normal'))
~/vid2speech/code/train.py:71: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(32, (3, 3), padding="same", kernel_initializer="he_normal")`
  model.add(Convolution2D(32, 3, 3, border_mode='same', init='he_normal'))
~/vid2speech/code/train.py:76: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(64, (3, 3), padding="same", kernel_initializer="he_normal")`
  model.add(Convolution2D(64, 3, 3, border_mode='same', init='he_normal'))
~/vid2speech/code/train.py:79: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(64, (3, 3), padding="same", kernel_initializer="he_normal")`
  model.add(Convolution2D(64, 3, 3, border_mode='same', init='he_normal'))
~/vid2speech/code/train.py:84: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(128, (3, 3), padding="same", kernel_initializer="he_normal")`
  model.add(Convolution2D(128, 3, 3, border_mode='same', init='he_normal'))
~/vid2speech/code/train.py:87: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(128, (3, 3), padding="same", kernel_initializer="he_normal")`
  model.add(Convolution2D(128, 3, 3, border_mode='same', init='he_normal'))
~/vid2speech/code/train.py:92: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(128, (3, 3), padding="same", kernel_initializer="he_normal")`
  model.add(Convolution2D(128, 3, 3, border_mode='same', init='he_normal'))
~/vid2speech/code/train.py:95: UserWarning: Update your `Conv2D` call to the Keras 2 API: `Conv2D(128, (3, 3), padding="same", kernel_initializer="he_normal")`
  model.add(Convolution2D(128, 3, 3, border_mode='same', init='he_normal'))
Traceback (most recent call last):
  File "predict.py", line 49, in <module>
    main()
  File "predict.py", line 42, in main
    model = train.build_model(net_out)
  File "~/vid2speech/code/train.py", line 100, in build_model
    model.add(Flatten())
  File "~/vid2speech/env/lib/python3.4/site-packages/keras/models.py", line 469, in add
    output_tensor = layer(self.outputs[0])
  File "~/vid2speech/env/lib/python3.4/site-packages/keras/engine/topology.py", line 615, in __call__
    output_shape = self.compute_output_shape(input_shape)
  File "~/vid2speech/env/lib/python3.4/site-packages/keras/layers/core.py", line 477, in compute_output_shape
    '(got ' + str(input_shape[1:]) + '. '
ValueError: The shape of the input to "Flatten" is not fully defined (got (0, 4, 128). Make sure to pass a complete "input_shape" or "batch_input_shape" argument to the first layer in your model.