Issue with testing

[clausmichele]

Hi, I was trying to test the model but I get the following errors:

Using TensorFlow backend. /home/cm72ein/Project/DeepMotion/deep-motion-master/src/FI_unet.py:59: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(32, (3, 3), padding="same", activation="relu") conv1 = Convolution2D(32, 3, 3, activation='relu', border_mode='same')(inputs) /home/cm72ein/Project/DeepMotion/deep-motion-master/src/FI_unet.py:61: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(32, (3, 3), padding="same", activation="relu") conv1 = Convolution2D(32, 3, 3, activation='relu', border_mode='same')(bn1) /home/cm72ein/Project/DeepMotion/deep-motion-master/src/FI_unet.py:65: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(64, (3, 3), padding="same", activation="relu") conv2 = Convolution2D(64, 3, 3, activation='relu', border_mode='same')(pool1) /home/cm72ein/Project/DeepMotion/deep-motion-master/src/FI_unet.py:67: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(64, (3, 3), padding="same", activation="relu") conv2 = Convolution2D(64, 3, 3, activation='relu', border_mode='same')(bn2) /home/cm72ein/Project/DeepMotion/deep-motion-master/src/FI_unet.py:71: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(128, (3, 3), padding="same", activation="relu") conv3 = Convolution2D(128, 3, 3, activation='relu', border_mode='same')(pool2) /home/cm72ein/Project/DeepMotion/deep-motion-master/src/FI_unet.py:73: UserWarning: Update your Conv2D call to the Keras 2 API: Conv2D(128, (3, 3), padding="same", activation="relu") conv3 = Convolution2D(128, 3, 3, activation='relu', border_mode='same')(bn3) Traceback (most recent call last): File "train.py", line 202, in main() File "train.py", line 85, in main model = get_unet_2(input_shape=(6, 128, 384)) File "/home/cm72ein/Project/DeepMotion/deep-motion-master/src/FI_unet.py", line 75, in get_unet_2 pool3 = MaxPooling2D(pool_size=(2, 2))(bn3) File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 619, in call output = self.call(inputs, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keras/layers/pooling.py", line 158, in call data_format=self.data_format) File "/usr/local/lib/python2.7/dist-packages/keras/layers/pooling.py", line 221, in _pooling_function pool_mode='max') File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 3657, in pool2d data_format=tf_data_format) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_ops.py", line 1958, in max_pool name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 2806, in _max_pool data_format=data_format, name=name) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper op_def=op_def) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2958, in create_op set_shapes_for_outputs(ret) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2209, in set_shapes_for_outputs shapes = shape_func(op) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2159, in call_with_requiring return call_cpp_shape_fn(op, require_shape_fn=True) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 627, in call_cpp_shape_fn require_shape_fn) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/common_shapes.py", line 691, in _call_cpp_shape_fn_impl raise ValueError(err.message) ValueError: Negative dimension size caused by subtracting 2 from 1 for 'max_pooling2d_3/MaxPool' (op: 'MaxPool') with input shapes: [?,1,32,128].

[lecode-official]

Same here, I have tried to install Keras 1.1.2 and Tensorflow 0.12.0 (0.10.0 is not available via PyPI anymore). Then I have tried to convert the code to the current Keras and Tensorflow versions:

inputs = Input((6, self.frame_height, self.frame_width))
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same', data_format="channels_first")(inputs)
bn1 = BatchNormalization()(conv1)
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn1)
bn1 = BatchNormalization()(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2), data_format="channels_first")(bn1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same', data_format="channels_first")(pool1)
bn2 = BatchNormalization()(conv2)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn2)
bn2 = BatchNormalization()(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2), data_format="channels_first")(bn2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_first")(pool2)
bn3 = BatchNormalization()(conv3)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn3)
bn3 = BatchNormalization()(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2), data_format="channels_first")(bn3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same', data_format="channels_first")(pool3)
bn4 = BatchNormalization()(conv4)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn4)
bn4 = BatchNormalization()(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2), data_format="channels_first")(bn4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same', data_format="channels_first")(pool4)
bn5 = BatchNormalization()(conv5)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn5)
bn5 = BatchNormalization()(conv5)
pool5 = MaxPooling2D(pool_size=(2, 2), data_format="channels_first")(bn5)
conv5_2 = Conv2D(512, (3, 3), activation='relu', padding='same', data_format="channels_first")(pool5)
bn5_2 = BatchNormalization()(conv5_2)
conv5_2 = Conv2D(512, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn5_2)
bn5_2 = BatchNormalization()(conv5_2)
up5_2 = concatenate([UpSampling2D(size=(2, 2))(bn5_2), bn5], axis=1)
conv6_2 = Conv2D(512, (3, 3), activation='relu', padding='same', data_format="channels_first")(up5_2)
bn6_2 = BatchNormalization()(conv6_2)
conv6_2 = Conv2D(512, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn6_2)
bn6_2 = BatchNormalization()(conv6_2)
up6 = concatenate([UpSampling2D(size=(2, 2))(bn6_2), bn4], axis=1)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same', data_format="channels_first")(up6)
bn6 = BatchNormalization()(conv6)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn6)
bn6 = BatchNormalization()(conv6)
up7 = concatenate([UpSampling2D(size=(2, 2))(bn6), bn3], axis=1)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_first")(up7)
bn7 = BatchNormalization()(conv7)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn7)
bn7 = BatchNormalization()(conv7)
up8 = concatenate([UpSampling2D(size=(2, 2))(bn7), bn2], axis=1)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same', data_format="channels_first")(up8)
bn8 = BatchNormalization()(conv8)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn8)
bn8 = BatchNormalization()(conv8)
up9 = concatenate([UpSampling2D(size=(2, 2))(bn8), bn1], axis=1)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same', data_format="channels_first")(up9)
bn9 = BatchNormalization()(conv9)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same', data_format="channels_first")(bn9)
bn9 = BatchNormalization()(conv9)
conv10 = Conv2D(3, (1, 1), activation='sigmoid')(bn9)

But that does not work either, because I always get to the problem, that the upscaling is incorrect (I am not very familar with Keras, so I guess some internal implementations have changed, which changed the dimensions of the tensors):

Traceback (most recent call last):
  File "/home/david/Repositories/tisr-masters-thesis/tisr/__main__.py", line 16, in <module>
    application.run()
  File "/home/david/Repositories/tisr-masters-thesis/tisr/application.py", line 89, in run
    command.run(command_line_arguments)
  File "/home/david/Repositories/tisr-masters-thesis/tisr/commands/predict_deep_motion_command.py", line 89, in run
    with InferenceModel(command_line_arguments.checkpoint_file_name, 384, 128) as inference_model:
  File "/home/david/Repositories/tisr-masters-thesis/tisr/neural_network/deep_motion_model.py", line 141, in __enter__
    self.initialize()
  File "/home/david/Repositories/tisr-masters-thesis/tisr/neural_network/deep_motion_model.py", line 88, in initialize
    up5_2 = concatenate([UpSampling2D(size=(2, 2))(bn5_2), bn5], axis=1)
  File "/usr/local/lib/python3.5/dist-packages/keras/layers/merge.py", line 641, in concatenate
    return Concatenate(axis=axis, **kwargs)(inputs)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/topology.py", line 594, in __call__
    self.build(input_shapes)
  File "/usr/local/lib/python3.5/dist-packages/keras/layers/merge.py", line 354, in build
    'Got inputs shapes: %s' % (input_shape))
ValueError: A `Concatenate` layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 1024, 8, 12), (None, 512, 8, 24)]

Any help will be greatly appreciated.

[lecode-official]

Okay, sorry, I just found the following comment: https://github.com/neil454/deep-motion/issues/2#issuecomment-283479980, which explains how to fix this. In that case the data_format as in my previous post, does not seem to be necessary. This is the code I used to successfully convert the code to work on Tensorflow 1.6.0 and Keras 2.1.6:

inputs = Input((6, self.frame_height, self.frame_width))
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)
bn1 = BatchNormalization()(conv1)
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(bn1)
bn1 = BatchNormalization()(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2))(bn1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(pool1)
bn2 = BatchNormalization()(conv2)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(bn2)
bn2 = BatchNormalization()(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(bn2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool2)
bn3 = BatchNormalization()(conv3)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(bn3)
bn3 = BatchNormalization()(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2))(bn3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(pool3)
bn4 = BatchNormalization()(conv4)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(bn4)
bn4 = BatchNormalization()(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2))(bn4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(pool4)
bn5 = BatchNormalization()(conv5)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(bn5)
bn5 = BatchNormalization()(conv5)
pool5 = MaxPooling2D(pool_size=(2, 2))(bn5)
conv5_2 = Conv2D(512, (3, 3), activation='relu', padding='same')(pool5)
bn5_2 = BatchNormalization()(conv5_2)
conv5_2 = Conv2D(512, (3, 3), activation='relu', padding='same')(bn5_2)
bn5_2 = BatchNormalization()(conv5_2)
up5_2 = concatenate([UpSampling2D(size=(2, 2))(bn5_2), bn5], axis=1)
conv6_2 = Conv2D(512, (3, 3), activation='relu', padding='same')(up5_2)
bn6_2 = BatchNormalization()(conv6_2)
conv6_2 = Conv2D(512, (3, 3), activation='relu', padding='same')(bn6_2)
bn6_2 = BatchNormalization()(conv6_2)
up6 = concatenate([UpSampling2D(size=(2, 2))(bn6_2), bn4], axis=1)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(up6)
bn6 = BatchNormalization()(conv6)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(bn6)
bn6 = BatchNormalization()(conv6)
up7 = concatenate([UpSampling2D(size=(2, 2))(bn6), bn3], axis=1)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(up7)
bn7 = BatchNormalization()(conv7)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(bn7)
bn7 = BatchNormalization()(conv7)
up8 = concatenate([UpSampling2D(size=(2, 2))(bn7), bn2], axis=1)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(up8)
bn8 = BatchNormalization()(conv8)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(bn8)
bn8 = BatchNormalization()(conv8)
up9 = concatenate([UpSampling2D(size=(2, 2))(bn8), bn1], axis=1)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(up9)
bn9 = BatchNormalization()(conv9)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(bn9)
bn9 = BatchNormalization()(conv9)
conv10 = Conv2D(3, (1, 1), activation='sigmoid')(bn9)

[clausmichele]

Can you expand on this? I copied and paste your code into the definition of get_unet_2but it still gives me an error:

ValueError: Negative dimension size caused by subtracting 2 from 1 for 'max_pooling2d_3/MaxPool' (op: 'MaxPool') with input shapes: [?,1,32,128].

[lecode-official]

Sorry, I should have elaborated a little more. The problem ist, that by default Keras uses TensorFlow channel ordering, i.e. height, width, channels. But in this case the author changed the channel ordering to Theano, i.e. channels, height, width. Neil probably set that in the Keras configuration file, so everyone checking out his code didn't have that configuration. But you can add the following two lines at the top of your code to switch the channel ordering just for this code:

import keras.backend
keras.backend.set_image_dim_ordering('th')

This should do the trick. Hope that helps.

[SpaceMonkeyForever]

@lecode-official I got fps_convert.py by following what you said, so thanks for that.

I'm downloading the pre-trained model because I can't get "train" to work and the data download script doesn't work any more because the KITTI data was moved on the server. I think I can skip training by using the pre-trained model.

The issue is that the model expects input with dimensions 384x128 and I want it to work with any video file. My understanding is it should work fine because the weights don't depend on dims of the input since it's a conv net (the same mask is applied regardless of the input size). However, Keras doesn't accept that. Is there any way around it?

To be clear, I'm doing:

model = get_unet_2((6, height, width))
model.load_weights("./../model_weights/weights_unet2_finetune_youtube_100epochs.hdf5") # throws error

where "height" and "width" are taken from the file I'm reading and are not equal to 384x128.

I get the following error:

ValueError: Dimension 0 in both shapes must be equal, but are 192 and 384. Shapes are [192] and [384]. for 'Assign_2' (op: 'Assign') with input shapes: [192], [384].

192 is the width I'm giving to get_unet_2 which doesn't match 384.

[SpaceMonkeyForever]

@lecode-official I see now that they mentioned the input dimensions cannot be changed because of batch normalisation in the model. I don't understand why this matters tho. How does batch normalisation on the hidden layers tie the dimensions of the input and the weights trained together?