Can't convert TF2 Zoo model

fotiDim commented 4 years ago

🐞Description the bug

Tensorflow model zoo was recently updated to TF2. I attempted to convert one of their sample models (efficientdet_d5_coco17_tpu-32) but got a model format error.

Trace

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
 in 
      4 
      5 # Convert to CoreML
----> 6 mlmodel = ct.convert(detect_fn, source='tensorflow')

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, **kwargs)
    256             outputs=outputs,
    257             classifier_config=classifier_config,
--> 258             **kwargs
    259         )
    260 

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in _convert(model, convert_from, convert_to, converter_registry, **kwargs)
    118     backend_converter = backend_converter_type()
    119 
--> 120     prog = frontend_converter(model, **kwargs)
    121     common_pass(prog)
    122     out = backend_converter(prog, **kwargs)

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in __call__(self, *args, **kwargs)
     50 
     51         tf2_loader = TF2Loader(*args, **kwargs)
---> 52         return tf2_loader.load()
     53 
     54 

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py in load(self)
     56         logging.info("Loading TensorFlow model '{}'".format(self.model))
     57         outputs = self.kwargs.get("outputs", None)
---> 58         self._graph_def = self._graph_def_from_model(outputs)
     59 
     60         if self._graph_def is not None and len(self._graph_def.node) == 0:

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py in _graph_def_from_model(self, outputs)
    108             return self.extract_sub_graph(graph_def, outputs)
    109         else:
--> 110             raise NotImplementedError(msg.format(self.model))
    111 
    112     def _tf_ssa_from_graph_def(self, fn_name="main"):

NotImplementedError: Expected model format: [SavedModel | [concrete_function] | tf.keras.Model | .h5], got ._UserObject object at 0x1852730d0>

To Reproduce

Use this notebook and add this code a last step:

!pip install coremltools==4.0b2
import coremltools as ct

# Convert to CoreML
mlmodel = ct.convert(detect_fn, source='tensorflow')

System environment (please complete the following information):

coremltools version 4.0b2
macOS version 10.15.5 (19F101)
XCode version (if applicable): Version 11.6 (11E708)
How you install python (anaconda, virtualenv, system): Pipenv This is my pipfile:
```
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
```

[dev-packages]

[packages] coremltools = "==4.0b2" tensorflow = "" jupyter = "" matplotlib = "*" scipy = "==1.4.1"

[requires] python_version = "3.7"

siddhantsomani commented 4 years ago

Can you try passing in the path to the SavedModel folder directly?

mlmodel = ct.convert(model="models/research/object_detection/test_data/efficientdet_d5_coco17_tpu-32/saved_model/", source="tensorflow")

fotiDim commented 4 years ago

@siddhantsomani in that case I am getting this trace:

Running TensorFlow Graph Passes:   0%|          | 0/5 [00:00<?, ? passes/s]ERROR:root:Constant Propagation pass failed: Fetch argument 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' cannot be interpreted as a Tensor. ("The name 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' refers to a Tensor which does not exist. The operation, 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert', exists but only has 0 outputs.")
Traceback (most recent call last):
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3556, in _as_graph_element_locked
    return op.outputs[out_n]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 305, in __init__
    fetch, allow_tensor=True, allow_operation=True))
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3512, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3561, in _as_graph_element_locked
    (repr(name), repr(op_name), len(op.outputs)))
KeyError: "The name 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' refers to a Tensor which does not exist. The operation, 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert', exists but only has 0 outputs."

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/tf_graph_pass/constant_propagation.py", line 96, in _constant_propagation
    result_list = sess.run(query_list)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 958, in run
    run_metadata_ptr)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1166, in _run
    self._graph, fetches, feed_dict_tensor, feed_handles=feed_handles)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 477, in __init__
    self._fetch_mapper = _FetchMapper.for_fetch(fetches)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 266, in for_fetch
    return _ListFetchMapper(fetch)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 378, in __init__
    self._mappers = [_FetchMapper.for_fetch(fetch) for fetch in fetches]
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 378, in <listcomp>
    self._mappers = [_FetchMapper.for_fetch(fetch) for fetch in fetches]
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 276, in for_fetch
    return _ElementFetchMapper(fetches, contraction_fn)
  File "/Users/fotidim/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 315, in __init__
    'Tensor. (%s)' % (fetch, str(e)))
ValueError: Fetch argument 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' cannot be interpreted as a Tensor. ("The name 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert:0' refers to a Tensor which does not exist. The operation, 'StatefulPartitionedCall/MultiscaleGridAnchorGenerator/GridAnchorGenerator/assert_equal_1/Assert/Assert', exists but only has 0 outputs.")
Running TensorFlow Graph Passes:  20%|██        | 1/5 [00:05<00:23,  5.94s/ passes]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-12-caac80a09913> in <module>
      5 # Convert to CoreML
      6 # mlmodel = ct.convert(detect_fn, source='tensorflow')
----> 7 mlmodel = ct.convert(model="models/research/object_detection/test_data/efficientdet_d5_coco17_tpu-32/saved_model/", source="tensorflow")

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, **kwargs)
    256             outputs=outputs,
    257             classifier_config=classifier_config,
--> 258             **kwargs
    259         )
    260 

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in _convert(model, convert_from, convert_to, converter_registry, **kwargs)
    118     backend_converter = backend_converter_type()
    119 
--> 120     prog = frontend_converter(model, **kwargs)
    121     common_pass(prog)
    122     out = backend_converter(prog, **kwargs)

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in __call__(self, *args, **kwargs)
     50 
     51         tf2_loader = TF2Loader(*args, **kwargs)
---> 52         return tf2_loader.load()
     53 
     54 

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py in load(self)
     77             )
     78 
---> 79         program = self._program_from_tf_ssa()
     80         logging.debug("program:\n{}".format(program))
     81         return program

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py in _program_from_tf_ssa(self)
    166                 tf_passes, desc="Running TensorFlow Graph Passes", unit=" passes"
    167             ):
--> 168                 tf_pass(self._tf_ssa)
    169 
    170         if self.debug:

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/tf_graph_pass/rewrite_control_flow_functions.py in rewrite_control_flow_functions(tf_ssa)
    543 def rewrite_control_flow_functions(tf_ssa):
    544     for fn_name, fn in tf_ssa.functions.items():
--> 545         _rewrite_cond_functions(tf_ssa, fn)
    546     for fn_name, fn in tf_ssa.functions.items():
    547         _eliminate_loop_cond_nodes(tf_ssa, fn)

~/.local/share/virtualenvs/le_models_ios/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/tf_graph_pass/rewrite_control_flow_functions.py in _rewrite_cond_functions(tf_ssa, fn)
    258                     idx = then_fn.outputs.index(mapped_name)
    259                 else:  # in else_fn.outputs
--> 260                     idx = else_fn.outputs.index(mapped_name)
    261             else:
    262                 idx = i

ValueError: '' is not in list

siddhantsomani commented 4 years ago

I can reproduce this. Any ideas if this is related to the 'delete_asserts control flow' bug @1duo ?

akhilraja-p commented 3 years ago

I have the same error. Unable to convert any tf2.0 models

akhilraja-p commented 3 years ago

++ @1duo for more visibility @jakesabathia2 This is similar to Issue

advaza commented 3 years ago

@fotiDim upgrading coremltools to 4.0b4 solved the issue to me. I'm trying to convert the same efficientdet model, but I'm now getting a different error:

Running TensorFlow Graph Passes: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00,  3.46 passes/s]
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-473b9a27a6e0> in <module>
----> 1 mlmodel = ct.convert(pb_path, source='tensorflow')

~/anaconda/lib/python3.6/site-packages/coremltools/converters/_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, **kwargs)
    263             outputs=outputs,
    264             classifier_config=classifier_config,
--> 265             **kwargs
    266         )
    267

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/converter.py in _convert(model, convert_from, convert_to, converter_registry, **kwargs)
    132     frontend_converter = frontend_converter_type()
    133
--> 134     prog = frontend_converter(model, **kwargs)
    135     common_pass(prog)
    136

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/converter.py in __call__(self, *args, **kwargs)
     72
     73         tf2_loader = TF2Loader(*args, **kwargs)
---> 74         return tf2_loader.load()
     75
     76

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py in load(self)
     78             )
     79
---> 80         program = self._program_from_tf_ssa()
     81         logging.debug("program:\n{}".format(program))
     82         return program

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py in _program_from_tf_ssa(self)
    176
    177         converter = TF2Converter(self._tf_ssa, **self.kwargs)
--> 178         return converter.convert()
    179
    180     def _populate_sub_graph_input_shapes(self, graph, graph_fns):

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py in convert(self)
    403         for g_name in self.graph_stack[1:]:
    404             self.context.add_graph(g_name, self.tfssa.functions[g_name].graph)
--> 405         self.convert_main_graph(prog, graph)
    406
    407         # Apply TF frontend passes on Program. These passes are different

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py in convert_main_graph(self, prog, graph)
    332             for name in func_inputs.keys():
    333                 self.context.add(name, ssa_func.inputs[name])
--> 334             outputs = convert_graph(self.context, graph, self.outputs)
    335             ssa_func.set_outputs(outputs)
    336             prog.add_function("main", ssa_func)

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py in convert_graph(context, graph, outputs)
    144     """
    145     connect_global_initializer(graph)
--> 146     nodes = topsort(graph)
    147
    148     if outputs is None:

~/anaconda/lib/python3.6/site-packages/coremltools/converters/mil/frontend/tensorflow/basic_graph_ops.py in topsort(graph)
    332         nextboundary = []
    333     if len(ret) != len(graph):
--> 334         raise ValueError("Graph is not a DAG!")
    335     return ret
    336

ValueError: Graph is not a DAG!

Any idea if that could be solved?

akhilraja-p commented 3 years ago

Same error for F-RCNN graphs as well. However for the SSD I get this error

Running TensorFlow Graph Passes: 100%|██████████| 5/5 [00:03<00:00, 1.47 passes/s] Converting Frontend ==> MIL Ops: 63%|██████▎ | 2847/4491 [00:05<00:03, 504.93 ops/s]

`NotImplementedError Traceback (most recent call last)

in 17 get_ipython().system('ls') 18 ---> 19 mlmodel = ct.convert("ssd/saved_model") 20 # mlmodel = ct.convert(tf_keras_model,source="tensorflow") 21 ~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, **kwargs) 263 outputs=outputs, 264 classifier_config=classifier_config, --> 265 **kwargs 266 ) 267 ~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in _convert(model, convert_from, convert_to, converter_registry, **kwargs) 132 frontend_converter = frontend_converter_type() 133 --> 134 prog = frontend_converter(model, **kwargs) 135 common_pass(prog) 136 ~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/converter.py in __call__(self, *args, **kwargs) 72 73 tf2_loader = TF2Loader(*args, **kwargs) ---> 74 return tf2_loader.load() 75 76 ~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py in load(self) 78 ) 79 ---> 80 program = self._program_from_tf_ssa() 81 logging.debug("program:\n{}".format(program)) 82 return program ~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py in _program_from_tf_ssa(self) 176 177 converter = TF2Converter(self._tf_ssa, **self.kwargs) --> 178 return converter.convert() 179 180 def _populate_sub_graph_input_shapes(self, graph, graph_fns): ~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py in convert(self) 403 for g_name in self.graph_stack[1:]: 404 self.context.add_graph(g_name, self.tfssa.functions[g_name].graph) --> 405 self.convert_main_graph(prog, graph) 406 407 # Apply TF frontend passes on Program. These passes are different ~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py in convert_main_graph(self, prog, graph) 332 for name in func_inputs.keys(): 333 self.context.add(name, ssa_func.inputs[name]) --> 334 outputs = convert_graph(self.context, graph, self.outputs) 335 ssa_func.set_outputs(outputs) 336 prog.add_function("main", ssa_func) ~/opt/anaconda3/envs/MLG/lib/python3.7/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py in convert_graph(context, graph, outputs) 178 node.op, node.original_node 179 ) --> 180 raise NotImplementedError(msg) 181 _add_op(context, node) 182 NotImplementedError: Conversion for TF op 'NonMaxSuppressionV5' not implemented. name: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/NonMaxSuppressionV5" op: "NonMaxSuppressionV5" input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/unstack" input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Reshape" input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/Minimum" input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/iou_threshold" input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/score_threshold" input: "StatefulPartitionedCall/Postprocessor/BatchMultiClassNonMaxSuppression/MultiClassNonMaxSuppression/non_max_suppression_with_scores/soft_nms_sigma" attr { key: "T" value { type: DT_FLOAT } } attr { key: "pad_to_max_output_size" value { b: false } }`

hlloydt commented 3 years ago

I am also seeing both these errors when converting models for the TensorFlow2 Zoo:

Converting RetinaNet: "NotImplementedError: Conversion for TF op 'NonMaxSuppressionV5' not implemented."
Converting EfficientDet: "ValueError: Graph is not a DAG!"

I would love to see these get fixed and the op 'NonMaxSuppressionV5' get implemented, please.

rafallukasik123 commented 3 years ago

I also have error "ValueError: Graph is not a DAG!" when i convert efficientDet from tensorflow repo. Efficientdet from automl repo throws error "Only a single concrete function is supported".

FruityLeo commented 3 years ago

I am also getting same errors "NotImplementedError: Conversion for TF op 'NonMaxSuppressionV5' not implemented." when trying to convert models like SSD MobileNet V1/V2 or SSD ResNet50.

Any update on this issue? As I would like to use TF Object Detection API for training a custom object detector and using it with CoreML. (Or is that already possible and I didn't find any further resources about this?)

cnmckee commented 3 years ago

Echo the above request: NonMaxSuppressionV5 would be very helpful.

kenlee-ga commented 3 years ago

Also encountered ValueError: Graph is not a DAG! while converting EfficientDet-d0 from TF2 to CoreML.

apple / coremltools