Cannot get converted keras model to run

[meiler]

I'm trying to get a converted Keras model to work with the neat CoreML semantic segmentation feature. I've tried a couple of different output formats, but keep getting the same error when applying the model in RectLabel:

Can you help me with what format is expected for the model to output, and how the meta data needs to be set for the model to play nicely with RectLabel?

Thank you very much. I've done a short write-up below of what I tried, and how to replicate.

How to replicate (update: solved, see solution in comment below):

I tried to create a small example of a converted Keras model. Normally I'd have a tensor with classes x height x width. And following your documentation, I add classes as user defined metadata

from keras.models import Sequential
from keras.layers import Conv2D
import coremltools

input_shape=[512, 512, 3]

# Create simple keras model
keras_model = Sequential()
keras_model.add(Conv2D(10, kernel_size=3, input_shape=input_shape, padding='same'))
keras_model.add(Conv2D(10, kernel_size=3, padding='same'))
keras_model.add(Conv2D(2, kernel_size=1, padding='same', activation='sigmoid'))
keras_model.compile(optimizer='sgd', loss='binary_crossentropy')

# Convert to coreml
coreml_model = coremltools.converters.keras.convert(
    model=keras_model,
    input_names='image',
    image_input_names='image',
    output_names=['output'],
    input_name_shape_dict={'image': keras_model.input.shape.as_list()},
    output_shapes=keras_model.output.shape.as_list()[1:-1],
    use_float_arraytype=True,
)

labels = ["person", "bicycle"] # dummy labels
coreml_model.user_defined_metadata['classes'] = ",".join(labels)

coreml_model.save('minimal_example.mlmodel')

The model loads in RectLabel, but raises the above undefined error.

I got the the DeepLabV3 model from apple to work, and have then tried to make my model match the one they use. Specifically their output is a height x width matrix, of integers that correspond to the class index. Therefore I tried to change the CoreML model after conversion, adding an ArgMax layer, to follow this pattern:

# Add some more metadata to the spec, to see if the the output not
# being set to integer is the problem

spec = coreml_model.get_spec()
INT32 = coremltools.proto.FeatureTypes_pb2.ArrayFeatureType.INT32
spec.description.output[0].type.multiArrayType.dataType = INT32
coremltools.utils.convert_double_to_float_multiarray_type(spec)

# This model also doesn't work
coremltools.utils.save_spec(spec, 'minimal_example_spec.mlmodel')

# Try adding an argmax layer in coreml.
spec = coremltools.models.utils.load_spec("minimal_example_spec.mlmodel")
builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec)

# To do that you have to remove the current output, and add a new layer
del spec.description.output[0]

builder.add_argmax('FinalArgMax',
                   input_name='output',
                   output_name='new_output',
                   axis=2,
                   )

new_output_layer = spec.description.output.add()

new_output_layer.name = 'new_output'
new_output_layer.shortDescription = 'restructure to classes'

new_output_params = new_output_layer.type.multiArrayType
new_output_params.dataType = coremltools.proto.FeatureTypes_pb2.ArrayFeatureType.ArrayDataType.Value(
    'INT32')
new_output_params.shape.extend(input_shape[:-1])
coremltools.utils.save_spec(spec, 'minimal_example_with_argmax.mlmodel')

But applying the model gives the same problem.

[ryouchinsa]

Thanks for writing the issue.

Using your small example of a converted Keras model, we could reproduce the problem.

If the model is DeepLab, specify in the short description when you convert.

coreml_model.short_description = "DeepLab"

If the model is DeepLab, the first label should be "background".

labels = ["background", "person", "bicycle"] 

The output shape is [height, width]. https://github.com/tucan9389/SemanticSegmentation-CoreML

We updated our document. https://rectlabel.com/help#load_coreml

Let us know your feedback.

[meiler]

Thank you! That worked. I assume that I can generally expect short_description to configure how RectLabel interprets the output?

Here's my new smallest, working 🎉, example (for anyone finding this):

from keras.models import Sequential
from keras.layers import Conv2D
import coremltools
from PIL import Image

input_shape=[512, 512, 3]

# Create simple keras model
keras_model = Sequential()
keras_model.add(Conv2D(10, kernel_size=3, input_shape=input_shape, padding='same'))
keras_model.add(Conv2D(10, kernel_size=3, padding='same'))
keras_model.add(Conv2D(2, kernel_size=1, padding='same', activation='softmax'))
keras_model.compile(optimizer='sgd', loss='binary_crossentropy')

# Convert to coreml
coreml_model = coremltools.converters.keras.convert(
    model=keras_model,
    input_names='image',
    image_input_names='image',
    output_names=['output'],
    input_name_shape_dict={'image': keras_model.input.shape.as_list()},
    output_shapes=keras_model.output.shape.as_list()[1:],
    use_float_arraytype=True,
)

labels = ["background", "bicycle"]
coreml_model.user_defined_metadata['classes'] = ",".join(labels)
coreml_model.short_description = "DeepLab"

# save most bare bones model
coreml_model.save('minimal_example.mlmodel')

# Create builder to add Argmax layer from CoreML
spec = coreml_model.get_spec()
builder = coremltools.models.neural_network.NeuralNetworkBuilder(spec=spec)

# Delete the current output node
del spec.description.output[0]
spec.neuralNetwork.layers[-1]

# Add a CoreML layer to do argmax on the labels dimension:
builder.add_argmax('FinalArgMax',
                   input_name='output',
                   output_name='semanticPredictions',
                   axis=2,
                   )

# Add a layer to the spec
semanticPredictions_layer = spec.description.output.add()

# Setup and configure references to the Layer build above
semanticPredictions_layer.name = 'semanticPredictions'
semanticPredictions_params = semanticPredictions_layer.type.multiArrayType
semanticPredictions_params.dataType = coremltools.proto.FeatureTypes_pb2.ArrayFeatureType.ArrayDataType.Value(
    'INT32')

# Note that this is needed to change the output shape from [1, 1, 1, 512, 512] to [512, 512]
semanticPredictions_params.shape.extend(input_shape[:-1])

# Save model
coremltools.utils.save_spec(spec, 'minimal_example_with_argmax.mlmodel')

Thanks so much for the quick reply!

[ryouchinsa]

Thanks for your feedback.

We are checking the short description to know the model output structure among below.

• Turi Create/Create ML
• YOLOv3/YOLOv3-tiny
• PoseNet
• DeepLab

If you think we should add the new output format of classes x height x width for semantic segmentation, let us know.

[meiler]

Thanks a lot - this works for now 😄