apple / coremltools

Core ML tools contain supporting tools for Core ML model conversion, editing, and validation.
https://coremltools.readme.io
BSD 3-Clause "New" or "Revised" License
4.32k stars 627 forks source link

Conversion from tensorflow with ImageProjectiveTransformV2 op is not working. #1426

Open maciej3031 opened 2 years ago

maciej3031 commented 2 years ago

🐞Describe the bug

When you have an ImageProjectiveTransformV2 op in tensorflow model and try to convert it to coreml then it raises the following exception. Is seems that affine ops does not support the "unnormalized" coordinates_mode. Do you plan to fix it in the near future? Affine transformation is quite commonly used in computer vision problems nowadays and it would be really nice to use it during conversion from tensorflow 🙏

Trace

Traceback (most recent call last):
  File "/Users/maciej3031/projects/coreml.py", line 257, in <module>
    mlmodel = ct.convert(model,
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/_converters_entry.py", line 352, in convert
    mlmodel = mil_convert(
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 183, in mil_convert
    return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, **kwargs)
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 210, in _mil_convert
    proto, mil_program = mil_convert_to_proto(
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 273, in mil_convert_to_proto
    prog = frontend_converter(model, **kwargs)
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 95, in __call__
    return tf2_loader.load()
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 84, in load
    program = self._program_from_tf_ssa()
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 200, in _program_from_tf_ssa
    return converter.convert()
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 401, in convert
    self.convert_main_graph(prog, graph)
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 330, in convert_main_graph
    outputs = convert_graph(self.context, graph, self.outputs)
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 189, in convert_graph
    add_op(context, node)
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/ops.py", line 1132, in ImageProjectiveTransformV2
    x = mb.affine(
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 63, in add_op
    return cls._add_op(op_cls, **kwargs)
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/mil/builder.py", line 191, in _add_op
    new_op.type_value_inference()
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/mil/operation.py", line 240, in type_value_inference
    output_types = self.type_inference()
  File "/Users/maciej3031/miniforge3/envs/facetracker-python/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/defs/image_resizing.py", line 125, in type_inference
    raise NotImplementedError(
NotImplementedError: input "coordinates_mode" to the "affine" not implemented. Got "unnormalized"

Minimal code to Reproduce

import os
import numpy as np
import tensorflow as tf
import coremltools as ct
from PIL import Image

img01 = np.random.rand(256, 256, 3).astype(np.float32)
img0255 = (img01*255).astype(np.uint8)
img_pil = Image.fromarray(img0255)

class ProblemLayer(tf.keras.layers.Layer):
    def __init__(self, **kwargs):
        super(ProblemLayer, self).__init__(**kwargs)

    def call(self, image):
        B, H, W = tf.shape(image)[0], tf.shape(image)[1], tf.shape(image)[2]
        proj_mat = tf.convert_to_tensor([[1.22, 0.09, -39.61, -0.09, 1.22, -6.17, 0.0, 0.0]], dtype=tf.float32)
        proj_mat = tf.repeat(proj_mat, B, 0)
        image = tf.raw_ops.ImageProjectiveTransformV2(
            images=image,
            transforms=proj_mat,
            output_shape=[H, W],
            interpolation='BILINEAR',
            fill_mode='CONSTANT')
        return image

inp = tf.keras.layers.Input(shape=(256, 256, 3), name="combined_input")
x = ProblemLayer()(inp)
model = tf.keras.models.Model(inputs=inp, outputs=x)

mlmodel = ct.convert(model,
                     inputs=[ct.ImageType(shape=(1, 256, 256, 3), scale=1.0/255.0)],
                     convert_to="mlprogram")

pred_coreml = mlmodel.predict({'combined_input': img_pil})

System environment

Additional context

Apart from this bug, do you plan to add support for ImageProjectiveTransformV3?

TobyRoseman commented 2 years ago

I'm not able to reproduce this problem. I get the following TensorFlow error by just declaring your class: AlreadyExistsError: Another metric with the same name already exists.

maciej3031 commented 2 years ago

Hey, that's very strange, everything works fine for me. To reproduce it once again I used the following commands:

  1. conda create -n test_env python=3.9
  2. conda activate test_env
  3. conda install tensorflow==2.6.0
  4. python3.9 -m pip install coremltools
  5. conda install pillow
  6. python3.9 coreml.py where coreml.py is:
    
    import os
    import numpy as np
    import tensorflow as tf
    import coremltools as ct
    from PIL import Image

img01 = np.random.rand(256, 256, 3).astype(np.float32) img0255 = (img01*255).astype(np.uint8) img_pil = Image.fromarray(img0255)

class ProblemLayer(tf.keras.layers.Layer): def init(self, kwargs): super(ProblemLayer, self).init(kwargs)

def call(self, image):
    B, H, W = tf.shape(image)[0], tf.shape(image)[1], tf.shape(image)[2]
    proj_mat = tf.convert_to_tensor([[1.22, 0.09, -39.61, -0.09, 1.22, -6.17, 0.0, 0.0]], dtype=tf.float32)
    proj_mat = tf.repeat(proj_mat, B, 0)
    image = tf.raw_ops.ImageProjectiveTransformV2(
        images=image,
        transforms=proj_mat,
        output_shape=[H, W],
        interpolation='BILINEAR',
        fill_mode='CONSTANT')
    return image

inp = tf.keras.layers.Input(shape=(256, 256, 3), name="combined_input") x = ProblemLayer()(inp) model = tf.keras.models.Model(inputs=inp, outputs=x)

mlmodel = ct.convert(model, inputs=[ct.ImageType(shape=(1, 256, 256, 3), scale=1.0/255.0)], convert_to="mlprogram")

pred_coreml = mlmodel.predict({'combined_input': img_pil})


and the error is still:

Traceback (most recent call last): File "/Users/maciej3031/projects/coreml.py", line 33, in mlmodel = ct.convert(model, File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/_converters_entry.py", line 352, in convert mlmodel = mil_convert( File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 183, in mil_convert return _mil_convert(model, convert_from, convert_to, ConverterRegistry, MLModel, compute_units, kwargs) File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 210, in _mil_convert proto, mil_program = mil_convert_to_proto( File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 273, in mil_convert_to_proto prog = frontend_converter(model, kwargs) File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/converter.py", line 95, in call return tf2_loader.load() File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/load.py", line 84, in load program = self._program_from_tf_ssa() File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow2/load.py", line 200, in _program_from_tf_ssa return converter.convert() File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 401, in convert self.convert_main_graph(prog, graph) File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/converter.py", line 330, in convert_main_graph outputs = convert_graph(self.context, graph, self.outputs) File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/convert_utils.py", line 189, in convert_graph add_op(context, node) File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/frontend/tensorflow/ops.py", line 1130, in ImageProjectiveTransformV2 x = mb.affine( File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/registry.py", line 63, in add_op return cls._add_op(op_cls, **kwargs) File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/mil/builder.py", line 191, in _add_op new_op.type_value_inference() File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/mil/operation.py", line 240, in type_value_inference output_types = self.type_inference() File "/Users/maciej3031/miniforge3/envs/test_env/lib/python3.9/site-packages/coremltools/converters/mil/mil/ops/defs/image_resizing.py", line 125, in type_inference raise NotImplementedError( NotImplementedError: input "coordinates_mode" to the "affine" not implemented. Got "unnormalized"

TobyRoseman commented 2 years ago

I think you're using non-standard conda channels. When running step 3, I get a PackagesNotFoundError.

If I follow your directions but install TensorFlow 2.6.0 using pip, I get the same error as before when declaring your class.

maciej3031 commented 2 years ago

I use mini conda forge from https://developer.apple.com/metal/tensorflow-plugin/ and tensorflow: conda-forge/osx-arm64::tensorflow-2.6.0-py39hb1906d9_1

maciej3031 commented 2 years ago

Did anyone succeed to reproduce it?

TobyRoseman commented 2 years ago

Ok, so you're on an M1 machine and using tensorflow-metal, those are important details.

Did anyone succeed to reproduce it?

Yes, using tensorflow-metal on an M1 machine, I can reproduce this problem.

maciej3031 commented 2 years ago

Yes, it's on M1. However, it occurs on both tensorflow-metal and usual tensorflow.