Convert SSD MobileNet V2 to CoreML

manozd commented 2 years ago

❓Question

I have an SSD MobileNet V2 model trained with Tensorflow Object detection API. How can I convert it to coreml? I tried the following steps but it did not work:

Convert checkpoints to saved model using exporter_main_v2.py
Load model model = tf.saved_model.load("path/to/model)
Run coremltools.convert(model, source="tensorflow")

I got an error: NotImplementedError: Expected model format: [SavedModel | [concrete_function] | tf.keras.Model | .h5], got <tensorflow.python.saved_model.load.Loader._recreate_base_user_object.<locals>._UserObject object at 0x7fab1a41cd90>

ubuntu 18 python version 3.7.6 Tensorflow version 2.8.2 coremltools version 5.2

TobyRoseman commented 2 years ago

What is the output of print(type(model))?

Relevant code section can be found here.

manozd commented 2 years ago

What is the output of print(type(model))?

Relevant code section can be found here.

It is the same as in error text above: <class 'tensorflow.python.saved_model.load.Loader._recreate_base_user_object.<locals>._UserObject'>

TobyRoseman commented 2 years ago

Can you give us a minimal example to reproduce the problem?

manozd commented 2 years ago

saved_model.zip

This is a model that i got after converting from checkpoints to saved_model using this. I also tried to convert it to tflite and its alright, so it is not problem with the model itself.

TobyRoseman commented 2 years ago

Loading an untrusted TensorFlow model is not secure. If you can give me a minimal example (i.e. something I just can copy and paste), I'll take a look. You could also dig into the relevant section of the code that I have already shared.

manozd commented 2 years ago

Loading an untrusted TensorFlow model is not secure. If you can give me a minimal example (i.e. something I just can copy and paste), I'll take a look. You could also dig into the relevant section of the code that I have already shared.

Download SSD MobileNet v2 320x320 model from trusted source

Run the following code:

import coremltools as ct
import tensorflow as tf

model = tf.saved_model.load("......./saved_model")
mlmodel = ct.convert(model, source="tensorflow")

manozd commented 2 years ago

I changed the code in function _get_concrete_functions_and_graph_def to following:

    def _get_concrete_functions_and_graph_def(self):
        saved_model = self.model
        sv = saved_model.signatures.values()
        cfs = sv if isinstance(sv, list) else list(sv)
        graph_def = self._graph_def_from_concrete_fn(cfs)

        return cfs, graph_def

And I got this error: Screenshot from 2022-06-29 14-14-47

TobyRoseman commented 2 years ago

I'm not sure what tf.saved_model.load is returning but looking at the dir of that object it certainly doesn't look like a TensorFlow model, i.e. doesn't seem like you can get predictions from it.

You need to pass ct.convert an actual TensorFlow model.

You said you managed to convert it to a tflite model, you could try converting that to Core ML. Is there anyway you can get a normal TensorFlow Model?

freedomtan commented 2 years ago

The ssd_mobilenet_v2_320x320_coco17_tpu-8/saved_model directory is a valid TensorFlow v2 (2.x) saved_model. We can get concrete_function from the loaded object easily.

If coremltools==6.0b1 is used

import tensorflow as tf
import coremltools as ct

m = tf.saved_model.load("/tmp/ssd_mobilenet_v2_320x320_coco17_tpu-8/saved_model")
f = m.__call__.get_concrete_function(tf.TensorSpec((1,320,320,3), tf.uint8))

input = ct.ImageType(name='input_tensor', shape=(1, 320, 320, 3))
ct.convert([f], "tensorflow", inputs=[input])

We can pass NonMaximumSuppressionV5 problem and get

Running TensorFlow Graph Passes: 100%|█████████████████████████████████████████████████████████████| 6/6 [00:01<00:00,  3.18 passes/s]
Converting TF Frontend ==> MIL Ops:  97%|████████████████████████████████████████████████████▎ | 4528/4671 [00:21<00:00, 210.51 ops/s]
Traceback (most recent call last):
  File "foobar.py", line 8, in <module>
    ct.convert([f], "tensorflow", inputs=[input])
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 426, in convert
    mlmodel = mil_convert(
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 182, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 209, in _mil_convert
    proto, mil_program = mil_convert_to_proto(
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 272, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 94, in __call__
    return tf2_loader.load()
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 86, in load
    program = self._program_from_tf_ssa()
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 200, in _program_from_tf_ssa
    return converter.convert()
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 473, in convert
    self.convert_main_graph(prog, graph)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 396, in convert_main_graph
    outputs = convert_graph(self.context, graph, self.output_names)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 189, in convert_graph
    add_op(context, node)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/ops.py", line 1391, in Cast
    raise NotImplementedError(
NotImplementedError: Cast: Provided destination type bool not supported.

if we fix the Cast with destination type bool issue, we have another problem

Running TensorFlow Graph Passes: 100%|█████████████████████████████████████████████████████████████| 6/6 [00:01<00:00,  3.15 passes/s]
Converting TF Frontend ==> MIL Ops:  97%|████████████████████████████████████████████████████▌ | 4551/4671 [00:22<00:00, 131.07 ops/s]WARNING:root:Saving value type of int64 into a builtin type of int32, might lose precision!
Converting TF Frontend ==> MIL Ops:  98%|████████████████████████████████████████████████████▋ | 4559/4671 [00:22<00:00, 198.34 ops/s]
Traceback (most recent call last):
  File "foobar.py", line 8, in <module>
    ct.convert([f], "tensorflow", inputs=[input])
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 426, in convert
    mlmodel = mil_convert(
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 182, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 209, in _mil_convert
    proto, mil_program = mil_convert_to_proto(
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 272, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 94, in __call__
    return tf2_loader.load()
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 86, in load
    program = self._program_from_tf_ssa()
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 200, in _program_from_tf_ssa
    return converter.convert()
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 473, in convert
    self.convert_main_graph(prog, graph)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 396, in convert_main_graph
    outputs = convert_graph(self.context, graph, self.output_names)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 189, in convert_graph
    add_op(context, node)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/ops.py", line 1948, in Select
    x = mb.select(cond=cond, a=a, b=b, name=node.name)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 63, in add_op
    return cls._add_op(op_cls, **kwargs)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/mil/builder.py", line 175, in _add_op
    new_op = op_cls(**kwargs)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/mil/ops/defs/control_flow.py", line 299, in __init__
    super().__init__(**kwargs)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/mil/operation.py", line 170, in __init__
    self._validate_and_set_inputs(input_kv)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/mil/operation.py", line 455, in _validate_and_set_inputs
    self.input_spec.validate_inputs(self.name, self.op_type, input_kvs)
  File "/Users/freedom/tf-master/lib/python3.8/site-packages/coremltools/converters/mil/mil/input_type.py", line 128, in validate_inputs
    raise ValueError(msg.format(name, var.name, input_type.type_str,
ValueError: Op "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Select_91" (op_type: select) Input cond="StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Greater" expects bool tensor but got bool

It seems to be some kind of tensor vs. scalar or broadcasting problem.

mbaharan commented 2 years ago

I have also installed coremltools==6.0b1 and received

Op "StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/linspace_3/SelectV2_1" (op_type: select) Input cond="StatefulPartitionedCall/StatefulPartitionedCall/StatefulPartitionedCall/linspace_3/GreaterEqual" expects bool tensor but got bool

for a different model.

freedomtan commented 2 years ago

FYR, currently, it seems if we want to have a whole TensorFlow SSD model (including NMS) converted and make it work in Xcode model preview, we have to do something like @hollance's "MobileNetV2 + SSDLite with Core ML". See my example of converting MobileDet, here.

Jeetendra-Shakya commented 2 years ago

Hi Toby, I tried to convert from ssdmobilenet to coreml using tfcoreml as well as coremltool but no luck. Can you please check my attachment and suggest what might be a problem? It is working fine when i convert to tflite for android devices. Thanks. SSD mobile net to core ml coversion.txt

TobyRoseman commented 2 years ago

@Jeetendra-Shakya - as I said earlier in this very issue loading an untrusted TensorFlow model is insecure.

Can someone please do some investigation and come up with a minimal example to reproduce the problem?

Jeetendra-Shakya commented 2 years ago

Hi Toby, Sorry I didn't understand what do you mean by "loading an untrusted TensorFlow model is insecure". It's my custom test model which is working fine when I created tflite model. I already sent you all steps i did in attachment if you want i can share you my model file as well. Please suggest.

apivovarov commented 2 years ago

I got NonMaxSuppressionV5 not implemented when converting TF2 SSD MobileNet v2 320x320 using coremltools 5.2 and 6.0b2

Example1 (convert saved_model dir):

import coremltools as ct
image_input = ct.ImageType(shape=(1, 320, 320, 3))
model = ct.convert("saved_model", inputs=[image_input])

Example2 (convert "serving_default" function)

import tensorflow as tf
m=tf.saved_model.load("saved_model")
f=m.signatures["serving_default"]
import coremltools as ct
image_input = ct.ImageType(shape=(1, 320, 320, 3))
model = ct.convert([f], inputs=[image_input])

Stack Trace when using version 5.2:

Converting Frontend ==> MIL Ops:  63%|███████████████████████████████████████████████████████████████████████████████████████████▉                                                     | 2847/4491 [00:04<00:02, 581.00 ops/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/_converters_entry.py", line 363, in convert
    debug=debug,
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/converter.py", line 183, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/converter.py", line 215, in _mil_convert
    **kwargs
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/converter.py", line 273, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/converter.py", line 95, in __call__
    return tf2_loader.load()
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 84, in load
    program = self._program_from_tf_ssa()
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 200, in _program_from_tf_ssa
    return converter.convert()
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 401, in convert
    self.convert_main_graph(prog, graph)
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 330, in convert_main_graph
    outputs = convert_graph(self.context, graph, self.outputs)
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 188, in convert_graph
    raise NotImplementedError(msg)
NotImplementedError: Conversion for TF op 'NonMaxSuppressionV5' not implemented.

name: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/NonMaxSuppressionV5"
op: "NonMaxSuppressionV5"

StackTrace when using version 6.0b2:

Converting TF Frontend ==> MIL Ops:  97%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋    | 4528/4671 [00:05<00:00, 765.76 ops/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/_converters_entry.py", line 463, in convert
    specification_version=specification_version,
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/converter.py", line 193, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/converter.py", line 225, in _mil_convert
    **kwargs
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/converter.py", line 283, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/converter.py", line 105, in __call__
    return tf2_loader.load()
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 86, in load
    program = self._program_from_tf_ssa()
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 208, in _program_from_tf_ssa
    return converter.convert()
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 475, in convert
    self.convert_main_graph(prog, graph)
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 398, in convert_main_graph
    outputs = convert_graph(self.context, graph, self.output_names)
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 189, in convert_graph
    add_op(context, node)
  File "/usr/local/lib/python3.7/dist-packages/coremltools/converters/mil/frontend/tensorflow/ops.py", line 1395, in Cast
    "supported.".format(types.get_type_info(node.attr["DstT"]))
NotImplementedError: Cast: Provided destination type bool not supported.

Jeetendra-Shakya commented 2 years ago

Guys, Is this Apple official github. If there is any problem converting custom model - I couldn't find any solution in net and they don't reply in developer site too. Is it like fix on your own. Very disappoint with such service from big organisation like APPLE.

apple / coremltools