search

AliaksandrSiarohin / first-order-model

This repository contains the source code for the paper First Order Motion Model for Image Animation
https://aliaksandrsiarohin.github.io/first-order-model-website/
MIT License
14.49k stars 3.21k forks source link

Is it possible to run first-order-model on mobile device? #382

Open la0216 opened 3 years ago

la0216 commented 3 years ago

I'm not sure whether I can run the first-order-model on mobile device, does anyone have suggestions on this? For example, compress the model, quantize the model or retrain the model, etc. Any suggesitons will be appreciated.

AliaksandrSiarohin commented 3 years ago

It is possible, because some people make an mobile app out of this. But I'm not an expert in runtime optimization, so could not tell. Everything that you suggest could be tried.

la0216 commented 3 years ago

Thank you so much for your feedback. Do you mean avatarify app?

AliaksandrSiarohin commented 3 years ago

For example, yes avatarify.

Noahz110 commented 3 years ago

You cant run this on a mobile device for the time being. Instead you can build a mobile app to call api to your server which do the rest.

sendy50 commented 3 years ago

ok but how i hosting this code i complete run this repository in windows but i don't know how to host on server Any suggesitons will be appreciated.

HashedViking commented 3 years ago

@AliaksandrSiarohin Are there some settings/arguments in generator, for example, that could be tweaked (even giving up the result quality) to boost the performance?

AliaksandrSiarohin commented 3 years ago

I never tried this, I guess tricks with replacing standard blocks with mobile net blocks should not hurt the performance much.

sendy50 commented 3 years ago

Okk but any flow for this ? I haven't any idea

On Mon, Mar 22, 2021, 3:05 PM AliaksandrSiarohin @.***> wrote:

I never tried this, I guess tricks with replacing standard blocks with mobile net blocks should not hurt the performance much.

β€” You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/AliaksandrSiarohin/first-order-model/issues/382#issuecomment-803914262, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDHS7HTR24IN22BOJVYBW3TE4FOHANCNFSM4YLFJMSA .

HashedViking commented 3 years ago

I never tried this, I guess tricks with replacing standard blocks with mobile net blocks should not hurt the performance much.

@AliaksandrSiarohin Thank you for a rapid reply. Could you, please, show some code snippets of how to use/replace those blocks? Is there anything else can be replaced/tweaked to reduce the computation complexity? May be some parameters like these:

  generator_params:
    block_expansion: 64
    max_features: 512
    num_down_blocks: 2
    num_bottleneck_blocks: 6
    estimate_occlusion_map: True
    dense_motion_params:
      block_expansion: 64
      max_features: 1024
      num_blocks: 5
      scale_factor: 0.25
HashedViking commented 3 years ago

Well, it is possible to run the model on mobile (iOS at least) but needs some optimisation. I was able to get 2 sec per frame on iPhone X Max.

urmit-space commented 3 years ago

Well, it is possible to run the model on mobile (iOS at least) but needs some optimisation. I was able to get 2 sec per frame on iPhone X Max.

Hey @HashedViking, can you please tell me how can do this? I'm finding a way to use this in iOS on the device. many stuff I do reverse engineering on the avatarify model and FOM model. they will use three models for this one is kp detector and jacobian series, encoder model, and another dense model. if you happy to share? thanks.

lshug commented 3 years ago

Some time ago I ported this architecture to Tensorflow. The resultant model is compatible with @AliaksandrSiarohin's checkpoints, and supports Tensorflow Lite. It can be used to generate tf lite files, which can then be used in mobile devices. However, the port is inference-only, so if you'd like to use your own trained model, you'll first have to train on the original pytorch version and transfer the checkpoints and config to tf version.

urmit-space commented 3 years ago

Some time ago I ported this architecture to Tensorflow. The resultant model is compatible with @AliaksandrSiarohin's checkpoints, and supports Tensorflow Lite. It can be used to generate tf lite files, which can then be used in mobile devices. However, the port is inference-only, so if you'd like to use your own trained model, you'll first have to train on the original pytorch version and transfer the checkpoints and config to tf version.

@lshug great work bro!!!! If I get time I will try to make and publish iOS mlmodel, if current engineering work.

HashedViking commented 3 years ago

@lshug magnificent work πŸš€ Have you tested it on a mobile device?

@urmit-space I was able to run the original model converted to TorchScript on mobile using LibTorch - only in CPU mode, because using Metal backend failed with errors like that: RuntimeError

Converting TorchScript model to CoreML is also failed due to missing ops like "grid_sample". Here is the implementation of grid_sample in pure Python by @OrkhanHI but it seems not very performant with triple nested for-loops (when i tried to torch.jit.trace the model using that implementation it ate about 30 GB of RAM in 2 hours and never finished even the first grid sampling) . @lshug's implementation of GridSample looks much more promising. We should check if Coremltools are capable to convert your tflite model to a native mlmodel.

urmit-space commented 3 years ago

We should check if Coremltools are capable to convert your tflite model to a native mlmodel.

@HashedViking we can't convert tflite to mlmodel. right process flow for better Metal performance is pth -> ONNX -> mlmodel. and we also need to optimize that layer by reducing the current one. like kp_detector model sample some layer are
"DivideBroadcastable" : 1, "Transpose" : 2, "ReshapeStatic" : 5, "LoadConstantND" : 2, "UpsampleNearestNeighbor" : 5, "Softmax" : 1, "ReduceSum" : 2, "Squeeze" : 1, "ExpandDims" : 3, "ActivationReLU" : 10, "PoolingAverage" : 5 i mean you got it what am i say. if you have good knowledge in optimization and can build model from giving layer information then let me know, we can do something in this.

lshug commented 3 years ago

@HashedViking I did test it on a mobile device, but the C++ code that used the generated tflite files for inference on-device was for a proprietary project so I can't share that part. However, I have the generated tflite files saved here. testlite.py shows a python example of using those files. Converting for your desired platform should be simple. Tensorflow documentation has guides on running tflite inference in C++ and Java for Android and in Swift for iOS.

@urmit-space coremltools can't convert from tflite, but it can convert from SavedModel directories, which my code generates along with tflite files.

urmit-space commented 3 years ago

@lshug, which intermediate model you had used? means your point that your save model directories have an intermediate model which can convert to mlmodel, am I correct?

HashedViking commented 3 years ago

@Ishug Unfortunately, only kp_detector SavedModel converted to CoreML model (same for PyTorch's kp_detector)

import coremltools as ct

name = 'kp_detector'
model = ct.convert(
    './saved_models/vox-adv/' + name,
    source="tensorflow"
)
model.save(name + '.mlmodel')
print('Done.')

converting the generator produces this error:

WARNING:root:TensorFlow version 2.4.1 detected. Last version known to be fully compatible is 2.3.1 .
Running TensorFlow Graph Passes: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 5/5 [00:04<00:00,  1.23 passes/s]
Converting Frontend ==> MIL Ops:  65%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ   | 583/895 [00:00<00:00, 1127.79 ops/s]
Traceback (most recent call last):
  File "/Users/SERG/development/first-order-model/first-order-model-tf-main/convert.py", line 4, in <module>
    model = ct.convert(
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 175, in convert
    mlmodel = mil_convert(
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 128, in mil_convert
    proto = mil_convert_to_proto(model, convert_from, convert_to,
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 171, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 75, in __call__
    return tf2_loader.load()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 79, in load
    program = self._program_from_tf_ssa()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 177, in _program_from_tf_ssa
    return converter.convert()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 406, in convert
    self.convert_main_graph(prog, graph)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 335, in convert_main_graph
    outputs = convert_graph(self.context, graph, self.outputs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 181, in convert_graph
    _add_op(context, node)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/ops.py", line 1829, in Select
    x = mb.select(cond=cond, a=a, b=b, name=node.name)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 62, in add_op
    return cls._add_op(op_cls, **kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/builder.py", line 174, in _add_op
    new_op = op_cls(**kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/ops/defs/control_flow.py", line 242, in __init__
    super(select, self).__init__(**kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/operation.py", line 170, in __init__
    self._validate_and_set_inputs(input_kv)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/operation.py", line 455, in _validate_and_set_inputs
    self.input_spec.validate_inputs(self.name, self.op_type, input_kvs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/input_type.py", line 120, in validate_inputs
    raise ValueError(msg.format(name, var.name, input_type.type_str,
ValueError: Op "StatefulPartitionedCall/model_1/deform/grid_sample/SelectV2" (op_type: select) Input b="StatefulPartitionedCall/model_1/deform/grid_sample/SelectV2/e" expects tensor but got int32

converting the process_kp_driving produces this error:

WARNING:root:TensorFlow version 2.4.1 detected. Last version known to be fully compatible is 2.3.1 .
Running TensorFlow Graph Passes: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 5/5 [00:01<00:00,  2.89 passes/s]
Converting Frontend ==> MIL Ops:  60%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ    | 6757/11195 [00:39<00:25, 172.00 ops/s]
Traceback (most recent call last):
  File "/Users/SERG/development/first-order-model/first-order-model-tf-main/convert.py", line 4, in <module>
    model = ct.convert(
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 175, in convert
    mlmodel = mil_convert(
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 128, in mil_convert
    proto = mil_convert_to_proto(model, convert_from, convert_to,
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 171, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 75, in __call__
    return tf2_loader.load()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 79, in load
    program = self._program_from_tf_ssa()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 177, in _program_from_tf_ssa
    return converter.convert()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 406, in convert
    self.convert_main_graph(prog, graph)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 335, in convert_main_graph
    outputs = convert_graph(self.context, graph, self.outputs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 180, in convert_graph
    raise NotImplementedError(msg)
NotImplementedError: Conversion for TF op 'MatrixDiagV3' not implemented.

name: "PartitionedCall/eye_2/diag"
op: "MatrixDiagV3"
input: "PartitionedCall/eye_2/ones"
input: "PartitionedCall/eye_2/diag/k"
input: "PartitionedCall/eye_2/diag/num_rows"
input: "PartitionedCall/eye_2/diag/num_cols"
input: "PartitionedCall/eye_2/diag/padding_value"
attr {
  key: "T"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "align"
  value {
    s: "RIGHT_LEFT"
  }
}

on the other hand - conversion of SavedModels and tflites to ONNX was successful for generator and kp_detector (interrupted process_kp_driving conversion after 40 mins of waiting) - but the following conversion to CoreML failed python -m tf2onnx.convert --saved-model "./saved_models/vox-adv/generator" --output generator.onnx

name = 'generator'
model = ct.converters.onnx.convert(model=name+'.onnx', minimum_ios_deployment_target='13')
model.save(name + '.mlmodel')
print('Done.')

Error: NotImplementedError: Unsupported ONNX ops of type: Loop

The last hope is to run tflite models with TensorFlow Lite for mobile.

HashedViking commented 3 years ago

@lshug What fps did you get on a mobile device? What are the device specs (iOS/Android, device model etc.)?

lshug commented 3 years ago

@HashedViking That's unfortunate. You could probably do without process_kp_driving: that module just implements @AliaksandrSiarohin's normalize_kp in-graph, and isn't very heavy. All the operations in there can just be done out-of-graph. On the other hand, looks like the conversion process for generator from savedmodel to mlmodel is a no-go, since the code depends heavily on gather_nd op in a lot of places.

Not sure about what's going on in the savedmodel>onnx>mlmodel pipeline for generator. There shouldn't be any loop-like ops in the graph for generator, otherwise tflite conversion would fail too. Could you paste the ONNX conversion log, so we can see at what part the Loop op creeps in?

I'm not sure about what fps we got in the end, as we weren't trying to run inference in real-time. I remember we tested on Galaxy S10e running LineageOS.

urmit-space commented 3 years ago

@HashedViking, I think we should change some layer which is supported by coremltools and onnx. and maybe fps is between 5-12. yes we can use tflite but hope all layer can be run on iOS, becoz some tflite model not supported by the iOS.

HashedViking commented 3 years ago

@lshug well here is my savedmodel>onnx>mlmodel pipeline: I'm using tensorflow-onnx converter

savedmodel>onnx

python -m tf2onnx.convert --saved-model "./saved_models/vox-adv/generator" --opset 9 --output generator.onnx --verbose (--opset 9 is default) Output:

/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/runpy.py:127: RuntimeWarning: 'tf2onnx.convert' found in sys.modules after import of package 'tf2onnx', but prior to execution of 'tf2onnx.convert'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))
2021-04-01 14:49:17,279 - WARNING - tf2onnx.tf_loader: '--tag' not specified for saved_model. Using --tag serve
2021-04-01 14:49:23,744 - INFO - tf2onnx.tf_loader: Signatures found in model: [serving_default].
2021-04-01 14:49:23,744 - WARNING - tf2onnx.tf_loader: '--signature_def' not specified, using first signature: serving_default
2021-04-01 14:49:23,745 - INFO - tf2onnx.tf_loader: Output names: ['output_0']
WARNING:tensorflow:From /Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/tf_loader.py:557: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2021-04-01 14:49:30,067 - WARNING - tensorflow: From /Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/tf_loader.py:557: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.compat.v1.graph_util.extract_sub_graph`
2021-04-01 14:49:34,702 - INFO - tf2onnx: inputs: ['kp_driving:0', 'kp_source:0', 'source_image:0']
2021-04-01 14:49:34,702 - INFO - tf2onnx: outputs: ['Identity:0']
2021-04-01 14:49:35,407 - INFO - tf2onnx.tfonnx: Using tensorflow=2.4.1, onnx=1.8.1, tf2onnx=1.8.4/cd55bf
2021-04-01 14:49:35,407 - INFO - tf2onnx.tfonnx: Using opset <onnx, 9>
2021-04-01 14:49:40,358 - INFO - tf2onnx.tf_utils: Computed 0 values for constant folding
2021-04-01 14:49:51,918 - VERBOSE - tf2onnx.tfonnx: Mapping TF node to ONNX node(s)
2021-04-01 14:50:05,691 - VERBOSE - tf2onnx.tfonnx: Summay Stats:
    tensorflow ops: Counter({'Const': 491, 'Conv2D': 33, 'BiasAdd': 30, 'FusedBatchNormV3': 27, 'Relu': 27, 'ExpandDims': 26, 'Mul': 25, 'ConcatV2': 24, 'StridedSlice': 22, 'Tile': 22, 'AddV2': 21, 'Sub': 17, 'Reshape': 16, 'GatherNd': 16, 'LogicalAnd': 16, 'SelectV2': 16, 'Shape': 15, 'Pack': 14, 'Cast': 12, 'Range': 9, 'Greater': 8, 'Less': 8, 'AvgPool': 7, 'Identity': 6, 'Sum': 5, 'Transpose': 5, 'Floor': 4, 'Placeholder': 3, 'Exp': 3, 'Pow': 2, 'Sigmoid': 2, 'NoOp': 1, 'Pad': 1, 'Max': 1, 'RealDiv': 1})
    tensorflow attr: Counter({'dtype': 988, 'value': 982, 'T': 776, 'data_format': 194, 'strides': 80, 'padding': 80, 'Tidx': 78, 'N': 76, 'use_cudnn_on_gpu': 66, 'explicit_paddings': 66, 'dilations': 66, 'exponential_avg_factor': 54, 'U': 54, 'is_training': 54, 'epsilon': 54, 'Tdim': 52, 'Index': 44, 'shrink_axis_mask': 44, 'ellipsis_mask': 44, 'begin_mask': 44, 'new_axis_mask': 44, 'end_mask': 44, 'Tmultiples': 44, 'Tshape': 32, 'Tparams': 32, 'Tindices': 32, 'out_type': 30, 'axis': 28, 'Truncate': 24, 'DstT': 24, 'SrcT': 24, 'ksize': 14, 'keep_dims': 12, 'Tperm': 10, 'shape': 6, 'Tpaddings': 2})
    onnx mapped: Counter({'Const': 456, 'Conv2D': 33, 'FusedBatchNormV3': 27, 'Relu': 27, 'ExpandDims': 26, 'Mul': 25, 'ConcatV2': 24, 'StridedSlice': 22, 'Tile': 22, 'AddV2': 21, 'Sub': 17, 'Reshape': 16, 'LogicalAnd': 16, 'SelectV2': 16, 'GatherNd': 16, 'Shape': 15, 'Pack': 14, 'Cast': 12, 'Range': 9, 'Less': 8, 'Greater': 8, 'AvgPool': 7, 'Identity': 6, 'Sum': 5, 'Transpose': 5, 'Floor': 4, 'Placeholder': 3, 'Exp': 3, 'Pow': 2, 'Sigmoid': 2, 'Pad': 1, 'Max': 1, 'RealDiv': 1})
    onnx unmapped: Counter()
2021-04-01 14:50:05,692 - INFO - tf2onnx.optimizer: Optimizing ONNX model
2021-04-01 14:50:05,697 - VERBOSE - tf2onnx.optimizer: Apply optimize_transpose
2021-04-01 14:50:06,163 - WARNING - tf2onnx.optimizer: Failed to apply optimize_transpose
Traceback (most recent call last):
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/__init__.py", line 57, in optimize_graph
    graph = opt.optimize(current) or graph
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/optimizer_base.py", line 41, in optimize
    graph = self._optimize(graph)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 143, in _optimize
    return self._apply_optimization(graph, self._optimize_at_current_graph_level)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/optimizer_base.py", line 62, in _apply_optimization
    graph = optimize_func(graph)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 156, in _optimize_at_current_graph_level
    if self._handle_nhwc_tranpose(n):
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 303, in _handle_nhwc_tranpose
    return op_handler(trans, p)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 781, in _slice_handler
    return self._switch_transpose_and_node(node, trans)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 284, in _switch_transpose_and_node
    new_shape = [shape[i] for i in NHWC_TO_NCHW]
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 284, in <listcomp>
    new_shape = [shape[i] for i in NHWC_TO_NCHW]
IndexError: list index out of range
2021-04-01 14:50:06,167 - VERBOSE - tf2onnx.optimizer: Apply remove_redundant_upsample
2021-04-01 14:50:06,542 - VERBOSE - tf2onnx.optimizer.UpsampleOptimizer: Const -144 (529->385)
2021-04-01 14:50:06,543 - VERBOSE - tf2onnx.optimizer: Apply fold_constants
2021-04-01 14:50:06,952 - VERBOSE - tf2onnx.optimizer.ConstFoldOptimizer: Cast -47 (152->105), Const +7 (385->392), Unsqueeze -47 (87->40)
2021-04-01 14:50:06,952 - VERBOSE - tf2onnx.optimizer: Apply const_dequantize_optimizer
2021-04-01 14:50:07,324 - VERBOSE - tf2onnx.optimizer.ConstDequantizeOptimizer: no change
2021-04-01 14:50:07,324 - VERBOSE - tf2onnx.optimizer: Apply loop_optimizer
2021-04-01 14:50:07,936 - VERBOSE - tf2onnx.optimizer.LoopOptimizer: no change
2021-04-01 14:50:07,936 - VERBOSE - tf2onnx.optimizer: Apply merge_duplication
2021-04-01 14:50:09,030 - VERBOSE - tf2onnx.optimizer.MergeDuplicatedNodesOptimizer: Cast -10 (105->95), Const -164 (392->228), Not -8 (16->8), Shape -2 (47->45), Slice -2 (86->84), Squeeze -2 (54->52), Transpose -1 (139->138), Unsqueeze -4 (40->36)
2021-04-01 14:50:09,030 - VERBOSE - tf2onnx.optimizer: Apply remove_identity
2021-04-01 14:50:09,397 - VERBOSE - tf2onnx.optimizer.IdentityOptimizer: Identity -32 (98->66)
2021-04-01 14:50:09,397 - VERBOSE - tf2onnx.optimizer: Apply remove_back_to_back
2021-04-01 14:50:10,529 - VERBOSE - tf2onnx.optimizer.BackToBackOptimizer: BatchNormalization -21 (27->6), Const -84 (228->144), Squeeze -11 (52->41), Transpose -44 (138->94), Unsqueeze -11 (36->25)
2021-04-01 14:50:10,529 - VERBOSE - tf2onnx.optimizer: Apply optimize_transpose
2021-04-01 14:50:10,792 - WARNING - tf2onnx.optimizer: Failed to apply optimize_transpose
Traceback (most recent call last):
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/__init__.py", line 57, in optimize_graph
    graph = opt.optimize(current) or graph
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/optimizer_base.py", line 41, in optimize
    graph = self._optimize(graph)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 143, in _optimize
    return self._apply_optimization(graph, self._optimize_at_current_graph_level)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/optimizer_base.py", line 62, in _apply_optimization
    graph = optimize_func(graph)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 156, in _optimize_at_current_graph_level
    if self._handle_nhwc_tranpose(n):
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 303, in _handle_nhwc_tranpose
    return op_handler(trans, p)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 781, in _slice_handler
    return self._switch_transpose_and_node(node, trans)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 284, in _switch_transpose_and_node
    new_shape = [shape[i] for i in NHWC_TO_NCHW]
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/tf2onnx/optimizer/transpose_optimizer.py", line 284, in <listcomp>
    new_shape = [shape[i] for i in NHWC_TO_NCHW]
IndexError: list index out of range
2021-04-01 14:50:10,793 - VERBOSE - tf2onnx.optimizer: Apply remove_redundant_upsample
2021-04-01 14:50:11,138 - VERBOSE - tf2onnx.optimizer.UpsampleOptimizer: no change
2021-04-01 14:50:11,138 - VERBOSE - tf2onnx.optimizer: Apply fold_constants
2021-04-01 14:50:11,889 - VERBOSE - tf2onnx.optimizer.ConstFoldOptimizer: no change
2021-04-01 14:50:11,889 - VERBOSE - tf2onnx.optimizer: Apply const_dequantize_optimizer
2021-04-01 14:50:12,229 - VERBOSE - tf2onnx.optimizer.ConstDequantizeOptimizer: no change
2021-04-01 14:50:12,229 - VERBOSE - tf2onnx.optimizer: Apply loop_optimizer
2021-04-01 14:50:12,578 - VERBOSE - tf2onnx.optimizer.LoopOptimizer: no change
2021-04-01 14:50:12,578 - VERBOSE - tf2onnx.optimizer: Apply merge_duplication
2021-04-01 14:50:13,084 - VERBOSE - tf2onnx.optimizer.MergeDuplicatedNodesOptimizer: no change
2021-04-01 14:50:13,084 - VERBOSE - tf2onnx.optimizer: Apply remove_identity
2021-04-01 14:50:13,424 - VERBOSE - tf2onnx.optimizer.IdentityOptimizer: no change
2021-04-01 14:50:13,425 - VERBOSE - tf2onnx.optimizer: Apply remove_back_to_back
2021-04-01 14:50:13,785 - VERBOSE - tf2onnx.optimizer.BackToBackOptimizer: no change
2021-04-01 14:50:13,803 - INFO - tf2onnx.optimizer: After optimization: BatchNormalization -21 (27->6), Cast -57 (152->95), Const -385 (529->144), Identity -32 (98->66), Not -8 (16->8), Shape -2 (47->45), Slice -2 (86->84), Squeeze -13 (54->41), Transpose -45 (139->94), Unsqueeze -62 (87->25)
2021-04-01 14:50:14,479 - INFO - tf2onnx:
2021-04-01 14:50:14,479 - INFO - tf2onnx: Successfully converted TensorFlow model ./saved_models/vox-adv/generator to ONNX
2021-04-01 14:50:14,479 - INFO - tf2onnx: Model inputs: ['kp_driving:0', 'kp_source:0', 'source_image:0']
2021-04-01 14:50:14,479 - INFO - tf2onnx: Model outputs: ['output_0']
2021-04-01 14:50:14,479 - INFO - tf2onnx: ONNX model is saved at generator.onnx

onnx>coreml

name = 'generator'
model = ct.converters.onnx.convert(model=name+'.onnx', minimum_ios_deployment_target='13')
model.save(name + '.mlmodel')
print('Done.')

Output:

WARNING:root:TensorFlow version 2.4.1 detected. Last version known to be fully compatible is 2.3.1 .
Traceback (most recent call last):
  File "/Users/SERG/development/first-order-model/first-order-model-tf-main/convert.py", line 12, in <module>
    model = ct.converters.onnx.convert(model=name+'.onnx', minimum_ios_deployment_target='13')
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/onnx/_converter.py", line 746, in convert
    _check_unsupported_ops(graph.nodes, disable_coreml_rank5_mapping)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/onnx/_converter.py", line 311, in _check_unsupported_ops
    raise NotImplementedError(
NotImplementedError: Unsupported ONNX ops of type: Loop

Also tflite conversion for generator failed until I added this in build.py:

generator_converter.target_spec.supported_ops = [
    tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
    tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
HashedViking commented 3 years ago

@lshug was the performance reasonable (at least 200ms per frame)? I'm just trying to figure out what way to head out - continue trying to convert something to pure CoreML or to try TF Lite on iOS.

lshug commented 3 years ago

@HashedViking I fixed the build issues. It should be able to build and shouldn't need SELECT_TF_OPS either. I fixed the input indexing issue in the tf lite test script too.

FPS-wise, I think it was processing around 2-3 frames per second on average. Not really sure though, and that was without any optimizations. I intend to write a test library for using the generated lite files and measuring performance on Android in a couple of days. I'll keep you updated.

zhoumenghan commented 3 years ago

@lshug Hi，thanks for your project！ I can do saved model β†’ onnx successfully. But when I continue transfer onnx to openvino，there are some errors： python mo_onnx.py --input_model E:\2D_Animation\first-order-model-tf-main\saved_models_float32\kp_detector.onnx --input img --input_shape [1,256,256,3] --output jacobian,value --output_dir E:\2D_Animation\first-order-model-tf-main\saved_models_float32\

Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      E:\2D_Animation\first-order-model-tf-main\saved_models_float32\kp_detector.onnx
        - Path for generated IR:        E:\2D_Animation\first-order-model-tf-main\saved_models_float32\
        - IR output name:       kp_detector
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         img
        - Output layers:        jacobian,value
        - Input shapes:         [1,256,256,3]
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       None
        - Reverse input channels:       False
ONNX specific parameters:
        - Inference Engine found in:    C:\Program Files (x86)\Intel\openvino_2021\python\python3.6\openvino
Inference Engine version:       2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Model Optimizer version:            2021.3.0-2787-60059f2c755-releases/2021/3
[ ERROR ]  -------------------------------------------------
[ ERROR ]  ----------------- INTERNAL ERROR ----------------
[ ERROR ]  Unexpected exception happened.
[ ERROR ]  Please contact Model Optimizer developers and forward the following information:
[ ERROR ]  Exception occurred during running replacer "REPLACEMENT_ID (<class 'extensions.middle.GatherNdNormalizer.GatherNDNormalize'>)": The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
[ ERROR ]  Traceback (most recent call last):
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\utils\class_registration.py", line 290, in apply_transform
    for_graph_and_each_sub_graph_recursively(graph, replacer.find_and_replace_pattern)
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\middle\pattern_match.py", line 59, in for_graph_and_each_sub_graph_recursively
    func(graph)
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\utils\replacement_pattern.py", line 27, in find_and_replace_pattern
    apply_pattern(graph, **self.pattern(), action=self.replace_pattern)  # pylint: disable=no-member
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\middle\pattern_match.py", line 96, in apply_pattern
    action(graph, match)
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\extensions\middle\GatherNdNormalizer.py", line 82, in replace_pattern
    gather_idx = self.indices_check(indices, input_shape)
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\extensions\middle\GatherNdNormalizer.py", line 62, in indices_check
    if not all(np.take(indices, indices=[i], axis=-1) == 0):
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\main.py", line 345, in main
    ret_code = driver(argv)
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\main.py", line 309, in driver
    ret_res = emit_ir(prepare_ir(argv), argv)
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\main.py", line 252, in prepare_ir
    graph = unified_pipeline(argv)
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\pipeline\unified.py", line 29, in unified_pipeline
    class_registration.ClassType.BACK_REPLACER
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\utils\class_registration.py", line 340, in apply_replacements
    apply_replacements_list(graph, replacers_order)
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\utils\class_registration.py", line 330, in apply_replacements_list
    num_transforms=len(replacers_order))
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\utils\logger.py", line 124, in wrapper
    function(*args, **kwargs)
  File "C:\Program Files (x86)\Intel\openvino_2021\deployment_tools\model_optimizer\mo\utils\class_registration.py", line 318, in apply_transform
    )) from err
Exception: Exception occurred during running replacer "REPLACEMENT_ID (<class 'extensions.middle.GatherNdNormalizer.GatherNDNormalize'>)": The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

[ ERROR ]  ---------------- END OF BUG REPORT --------------
[ ERROR ]  -------------------------------------------------

I donβ€˜t know what I can do to solve it，I need your help~please

lshug commented 3 years ago

@zhoumenghan There seems to be a bug in openvino model optimizer's GatherNdNormalizer.py. The line if not all(np.take(indices, indices=[i], axis=-1) == 0): should be if not np.array(np.take(indices, indices=[i], axis=-1) == 0).all().

If patching that file isn't an option, you might be able to avoid the problem by building with --staticbatchsize. When that's enabled the models will use tf.gather instead of tf.gather_nd.

zhoumenghan commented 3 years ago

@zhoumenghan There seems to be a bug in openvino model optimizer's GatherNdNormalizer.py. The line if not all(np.take(indices, indices=[i], axis=-1) == 0): should be if not np.array(np.take(indices, indices=[i], axis=-1) == 0).all().

If patching that file isn't an option, you might be able to avoid the problem by building with --staticbatchsize. When that's enabled the models will use tf.gather instead of tf.gather_nd.

@lshug thanks~ In fact, I also modified this way like The line if not all(np.take(indices, indices=[i], axis=-1) == 0): should be if not np.array(np.take(indices, indices=[i], axis=-1) == 0).all().yesterday, so that the openvino file was output correctly. I alse have another problem: when I use python mo_onnx.py --input_model E:\2D_Animation\first-order-model-tf-main\saved_models_float32\process_kp_driving.onnx --input adapt_movement_scale,kp_driving,kp_driving_initial,kp_driving_initial_jacobian,kp_driving_jacobian,kp_source,kp_source_jacobian,use_relative_jacobian --input_shape (),(1,10,2),(1,10,2),(1,10,2,2),(1,10,2,2),(1,10,2),(1,10,2,2),() --output jacobian,value --output_dir E:\2D_Animation\first-order-model-tf-main\saved_models_float32\onnx

Model Optimizer arguments:
Common parameters:
        - Path to the Input Model:      E:\2D_Animation\first-order-model-tf-main\saved_models_float32\process_kp_driving.onnx
        - Path for generated IR:        E:\2D_Animation\first-order-model-tf-main\saved_models_float32\onnx
        - IR output name:       process_kp_driving
        - Log level:    ERROR
        - Batch:        Not specified, inherited from the model
        - Input layers:         adapt_movement_scale,kp_driving,kp_driving_initial,kp_driving_initial_jacobian,kp_driving_jacobian,kp_source,kp_source_jacobian,use_relative_jacobian
        - Output layers:        jacobian,value
        - Input shapes:         (),(1,10,2),(1,10,2),(1,10,2,2),(1,10,2,2),(1,10,2),(1,10,2,2),()
        - Mean values:  Not specified
        - Scale values:         Not specified
        - Scale factor:         Not specified
        - Precision of IR:      FP32
        - Enable fusing:        True
        - Enable grouped convolutions fusing:   True
        - Move mean values to preprocess section:       None
        - Reverse input channels:       False
ONNX specific parameters:
        - Inference Engine found in:    C:\Program Files (x86)\Intel\openvino_2021\python\python3.6\openvino
Inference Engine version:       2.1.2021.3.0-2787-60059f2c755-releases/2021/3
Model Optimizer version:            2021.3.0-2787-60059f2c755-releases/2021/3
[ ERROR ]  Input shape "(),(1,10,2),(1,10,2),(1,10,2,2),(1,10,2,2),(1,10,2),(1,10,2,2),()" cannot be parsed.

to transfer onnx to openvino, I don't know how to describe the shape of adapt_movement_scale and use_relative_jacobian for transfering openvino，which are only a value，e.g. adapt_movement_scale=0.0 use_relative_jacobian=1.0

Looking forward to your reply~~~

lshug commented 3 years ago

@zhoumenghan adapt_movement_scale and use_relative_jacobian are scalar inputs. If you want them to be changeable during serving time (probably useless for use_relative_jacobian), then you'll have to modify the code to have them be single-element vectors instead, by replacing their shapes in signature with (1,) and accesses to their values to adapt_movement_scale[0] and use_relative_jacobian[0].

If you don't care about changing those values during serving time (which is probably the case; you'll likely always want use_relative_jacobian to be on and you'll probably either always or never do adapt_movement_scale), then instead you can just build with --hardcode and replace the shapes in signature with (1,), then pass dummy single-element vectors during inference.

RAJA-PARIKSHAT commented 2 years ago

@lshug Great work by you, actually my problem is similar to that of yours. I want to run the model on a CPU with at least 6 fps. I am trying to reduce the model size by dropping some layers and retraining in it on 5000 videos of the Vox dataset. I am using a batch size of 24, but I am getting weird results. What can be the possible issue? My application is simple transferring only smile, blink, or simple expressions from one face to another. A little guidance in this regard will be great. Thanks

MadeByKit commented 1 year ago

@ishug Unfortunately, only kp_detector SavedModel converted to CoreML model (same for PyTorch's kp_detector)

import coremltools as ct

name = 'kp_detector'
model = ct.convert(
    './saved_models/vox-adv/' + name,
    source="tensorflow"
)
model.save(name + '.mlmodel')
print('Done.')

converting the generator produces this error:

WARNING:root:TensorFlow version 2.4.1 detected. Last version known to be fully compatible is 2.3.1 .
Running TensorFlow Graph Passes: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 5/5 [00:04<00:00,  1.23 passes/s]
Converting Frontend ==> MIL Ops:  65%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ   | 583/895 [00:00<00:00, 1127.79 ops/s]
Traceback (most recent call last):
  File "/Users/SERG/development/first-order-model/first-order-model-tf-main/convert.py", line 4, in <module>
    model = ct.convert(
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 175, in convert
    mlmodel = mil_convert(
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 128, in mil_convert
    proto = mil_convert_to_proto(model, convert_from, convert_to,
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 171, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 75, in __call__
    return tf2_loader.load()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 79, in load
    program = self._program_from_tf_ssa()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 177, in _program_from_tf_ssa
    return converter.convert()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 406, in convert
    self.convert_main_graph(prog, graph)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 335, in convert_main_graph
    outputs = convert_graph(self.context, graph, self.outputs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 181, in convert_graph
    _add_op(context, node)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/ops.py", line 1829, in Select
    x = mb.select(cond=cond, a=a, b=b, name=node.name)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 62, in add_op
    return cls._add_op(op_cls, **kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/builder.py", line 174, in _add_op
    new_op = op_cls(**kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/ops/defs/control_flow.py", line 242, in __init__
    super(select, self).__init__(**kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/operation.py", line 170, in __init__
    self._validate_and_set_inputs(input_kv)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/operation.py", line 455, in _validate_and_set_inputs
    self.input_spec.validate_inputs(self.name, self.op_type, input_kvs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/mil/input_type.py", line 120, in validate_inputs
    raise ValueError(msg.format(name, var.name, input_type.type_str,
ValueError: Op "StatefulPartitionedCall/model_1/deform/grid_sample/SelectV2" (op_type: select) Input b="StatefulPartitionedCall/model_1/deform/grid_sample/SelectV2/e" expects tensor but got int32

converting the process_kp_driving produces this error:

WARNING:root:TensorFlow version 2.4.1 detected. Last version known to be fully compatible is 2.3.1 .
Running TensorFlow Graph Passes: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 5/5 [00:01<00:00,  2.89 passes/s]
Converting Frontend ==> MIL Ops:  60%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ    | 6757/11195 [00:39<00:25, 172.00 ops/s]
Traceback (most recent call last):
  File "/Users/SERG/development/first-order-model/first-order-model-tf-main/convert.py", line 4, in <module>
    model = ct.convert(
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py", line 175, in convert
    mlmodel = mil_convert(
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 128, in mil_convert
    proto = mil_convert_to_proto(model, convert_from, convert_to,
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 171, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/converter.py", line 75, in __call__
    return tf2_loader.load()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 79, in load
    program = self._program_from_tf_ssa()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 177, in _program_from_tf_ssa
    return converter.convert()
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 406, in convert
    self.convert_main_graph(prog, graph)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 335, in convert_main_graph
    outputs = convert_graph(self.context, graph, self.outputs)
  File "/Users/SERG/opt/anaconda3/envs/fom/lib/python3.8/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 180, in convert_graph
    raise NotImplementedError(msg)
NotImplementedError: Conversion for TF op 'MatrixDiagV3' not implemented.

name: "PartitionedCall/eye_2/diag"
op: "MatrixDiagV3"
input: "PartitionedCall/eye_2/ones"
input: "PartitionedCall/eye_2/diag/k"
input: "PartitionedCall/eye_2/diag/num_rows"
input: "PartitionedCall/eye_2/diag/num_cols"
input: "PartitionedCall/eye_2/diag/padding_value"
attr {
  key: "T"
  value {
    type: DT_FLOAT
  }
}
attr {
  key: "align"
  value {
    s: "RIGHT_LEFT"
  }
}

on the other hand - conversion of SavedModels and tflites to ONNX was successful for generator and kp_detector (interrupted process_kp_driving conversion after 40 mins of waiting) - but the following conversion to CoreML failed python -m tf2onnx.convert --saved-model "./saved_models/vox-adv/generator" --output generator.onnx

name = 'generator'
model = ct.converters.onnx.convert(model=name+'.onnx', minimum_ios_deployment_target='13')
model.save(name + '.mlmodel')
print('Done.')

Error: NotImplementedError: Unsupported ONNX ops of type: Loop

The last hope is to run tflite models with TensorFlow Lite for mobile.

please can you tell me how to get the saved model of generate and kp_detector?

Apokar commented 3 months ago

hello , May I ask do you build an app to run this model on mobile device? Can you share some details with me ,it will be very helpful, thanks!