LSTM & Conv2D Keras input shape issues

[teaglin]

Hi,

I am having issues exporting the following model to CoreML. Below is the model in Keras.

model = Sequential() model.add(TimeDistributed(Convolution2D(16, 3, 3, border_mode="same", activation='relu'), input_shape=(8, image_height, image_width, 3))) model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2)))) model.add(TimeDistributed(Convolution2D(32, 3, 3, activation='relu'))) model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2)))) model.add(TimeDistributed(Convolution2D(64, 3, 3, activation='relu'))) model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2)))) model.add(TimeDistributed(Convolution2D(128, 3, 3, activation='relu'))) model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2)))) model.add(TimeDistributed(Flatten())) model.add(LSTM(256, return_sequences=True)) model.add(TimeDistributed(Dense(1)))

CoreML Tools just says "ValueError: Input 'image' has input shape of length 4" Any ideas of what I need to do to fix this?

[aseemw]

Yes there is an issue with the converter currently. It does not handle "TimeDistributed()" functions properly.

In CoreML, there is always a sequence dimension (see documentation) So unlike Keras, there is no need to specify "TimeDistributed()" separately. The converter thus sort of ignores the "TimeDistributed()" keyword, but here the 4-D shape to a convolution (when it expects 3-D) is throwing the converter off.

A workaround would be to build a different Keras model for the purpose of conversion (using get_weights() and set_weights() functions you should be able to transfer the weights from the actual trained Keras model to this new one for conversion)

model = Sequential()
model.add(Convolution2D(16, 3, 3, border_mode="same", activation='relu', input_shape=(100, 100, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(32, 3, 3, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(64, 3, 3, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(128, 3, 3, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Reshape((1,-1))) #seq length = 1 for the purpose of conversion, at runtime, any length can be given
model.add(LSTM(256, return_sequences=True))
model.add(Dense(1))

[schliffen]

Well, How do you solve label issue? for some frames we have just one label

[schliffen]

I suggest train them separately and make two different mlmodels

[TobyRoseman]

Multi-backend Keras support has been removed in coremltools 6.0.