apple / coremltools

Core ML tools contain supporting tools for Core ML model conversion, editing, and validation.
https://coremltools.readme.io
BSD 3-Clause "New" or "Revised" License
4.41k stars 636 forks source link

LSTM & Conv2D Keras input shape issues #202

Closed teaglin closed 2 years ago

teaglin commented 6 years ago

Hi,

I am having issues exporting the following model to CoreML. Below is the model in Keras.

model = Sequential() model.add(TimeDistributed(Convolution2D(16, 3, 3, border_mode="same", activation='relu'), input_shape=(8, image_height, image_width, 3))) model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2)))) model.add(TimeDistributed(Convolution2D(32, 3, 3, activation='relu'))) model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2)))) model.add(TimeDistributed(Convolution2D(64, 3, 3, activation='relu'))) model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2)))) model.add(TimeDistributed(Convolution2D(128, 3, 3, activation='relu'))) model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2)))) model.add(TimeDistributed(Flatten())) model.add(LSTM(256, return_sequences=True)) model.add(TimeDistributed(Dense(1)))

CoreML Tools just says "ValueError: Input 'image' has input shape of length 4" Any ideas of what I need to do to fix this?

aseemw commented 6 years ago

Yes there is an issue with the converter currently. It does not handle "TimeDistributed()" functions properly.

In CoreML, there is always a sequence dimension (see documentation) So unlike Keras, there is no need to specify "TimeDistributed()" separately. The converter thus sort of ignores the "TimeDistributed()" keyword, but here the 4-D shape to a convolution (when it expects 3-D) is throwing the converter off.

A workaround would be to build a different Keras model for the purpose of conversion (using get_weights() and set_weights() functions you should be able to transfer the weights from the actual trained Keras model to this new one for conversion)

model = Sequential()
model.add(Convolution2D(16, 3, 3, border_mode="same", activation='relu', input_shape=(100, 100, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(32, 3, 3, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(64, 3, 3, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Convolution2D(128, 3, 3, activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Reshape((1,-1))) #seq length = 1 for the purpose of conversion, at runtime, any length can be given
model.add(LSTM(256, return_sequences=True))
model.add(Dense(1))
schliffen commented 6 years ago

Well, How do you solve label issue? for some frames we have just one label

schliffen commented 6 years ago

I suggest train them separately and make two different mlmodels

TobyRoseman commented 2 years ago

Multi-backend Keras support has been removed in coremltools 6.0.