BastinFlorian commented 11 months ago

🐞Describing the bug

I try to do a gesture recognition model using TF, custom tf function.
The custom functions are needed for data preprocessing and is done in TF because it is easier to communicate with the IOS dev team
I successfully compressed model as a ml package. But when I try to do inference I get an error.
It works when converting to TF Lite

Stack Trace

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[77], line 6
      4 # Load the model
      5 model = ct.models.MLModel(coreml_preprocess_path, compute_units=ct.ComputeUnit.CPU_ONLY)
----> 6 predictions = model.predict(input)

File ~/Documents/XXX/XXX/XXX/XXX/lib/python3.10/site-packages/coremltools/models/model.py:579, in MLModel.predict(self, data)
    576 MLModel._check_predict_data(data)
    578 if self.__proxy__:
--> 579     return MLModel._get_predictions(self.__proxy__, verify_and_convert_input_dict, data)
    580 else:   # Error case
    581     if _macos_version() < (10, 13):

File ~/Documents/XXX/XXX/XXX/XXX/lib/python3.10/site-packages/coremltools/models/model.py:631, in MLModel._get_predictions(proxy, preprocess_method, data)
    629 if type(data) == dict:
    630     preprocess_method(data)
--> 631     return proxy.predict(data)
    632 else:
    633     assert type(data) == list

RuntimeError: {
    NSLocalizedDescription = "Error in declaring network.";
}

To Reproduce

Please add a minimal code example that can reproduce the error when running it. OK


import os
import math
import numpy as np
import tensorflow as tf
import coremltools as ct
from importlib.metadata import version

@tf.function def tf_rotate_vertically(landmarks):

Compute vertical line

y = (landmarks[:, 9, 0] + landmarks[:, 13, 0]) / 2
x = (landmarks[:, 9, 1] + landmarks[:, 13, 1]) / 2

# Angle to substract
angle = tf.where(tf.less(x, 0.0), tf.atan(y / x) + np.pi, tf.atan(y / (x + 1e-12)))
angle = tf.where(tf.less(y, 0.0), angle - (2.0 * np.pi), angle)
angle = tf.where(tf.logical_and(tf.equal(x, 0.0), tf.equal(y, 0.0)), tf.zeros_like(x), angle)

# Get rotation angle radian to have vertical hand
angle_radian = float(math.pi / 2) - angle

# Create the rotation matric
rotation_matrix = tf.stack(
    [
        (tf.cos(angle_radian), -tf.sin(angle_radian)),
        (tf.sin(angle_radian), tf.cos(angle_radian))
    ],
    axis=0
)

rotation_matrix = tf.transpose(rotation_matrix)
landmarks_rotated = tf.linalg.matmul(landmarks, rotation_matrix)

return landmarks_rotated

@tf.function def tf_atan2(y, x): angle = tf.where(tf.less(x, 0.0), tf.atan(y / x) + np.pi, tf.atan(y / (x + 1e-12))) angle = tf.where(tf.less(y, 0.0), angle - (2.0 * np.pi), angle) angle = tf.where(tf.logical_and(tf.equal(x, 0.0), tf.equal(y, 0.0)), tf.zeros_like(x), angle) return angle

Keras model

Model inputs

input_scalar = tf.keras.layers.Input(shape=(1), dtype=tf.float32, name='handedness') input_matrix = tf.keras.layers.Input(shape=(21, 2), dtype=tf.float32, name='landmarks')

Keep first idx of each sample

first_element = tf.gather(input_matrix, 0, axis=1, name='first_element')

Substract by first element

x = tf.keras.layers.subtract([input_matrix, first_element], name='substract', input_shape=(21, 2))

Rotation

x = tf.keras.layers.Lambda(tf_rotate_vertically, name='rotate_vertically', input_shape=(21, 2))(x)

preprocessing_model = tf.keras.Model(inputs=input_matrix, outputs=x)

preprocessing_model.predict(tf.reshape(np.random.rand(10, 21, 2), [10, 21, 2])) # OK

Save model

Convert the model to Core ML

input_matrix_shape = ct.Shape(shape=(1, 21, 2))

coreml_model = ct.convert( preprocessing_model, inputs=[ ct.TensorType(name='landmarks', shape=input_matrix_shape), ], source='tensorflow', convert_to="mlprogram" )

coreml_preprocess_path = '../model/IOS/preprocess_model.mlpackage'

Save the model

coreml_model.save(coreml_preprocess_path)


**KO**
```python
# Random Input to test loaded ml
input = {
    "landmarks": np.random.rand(1, 21, 2),
}

# Load the model
model = ct.models.MLModel(coreml_preprocess_path, compute_units=ct.ComputeUnit.CPU_ONLY)
predictions = model.predict(input)

System environment (please complete the following information):

coremltools version: 7.0
numpy version: 1.24.3
tensorflow version: 2.13.0
- OS (e.g. MacOS version or Linux type): M1 Pro 13.6

BastinFlorian commented 11 months ago

Any idea ? @TobyRoseman

junpeiz commented 11 months ago

Thanks @BastinFlorian for providing the reproduce script. I confirmed that I can reproduce this error on my side.

junpeiz commented 11 months ago

It's an issue in CoreML, and I have filed an internal ticket to track it. Will keep you updated when it's fixed. Thanks!

junpeiz commented 10 months ago

Hey @BastinFlorian, if you want to try some quick workarounds on your side: This bug is introduced by indexing assignment, so you might be able to work it around if you could try to replace it with scatter op. Thanks!

BastinFlorian commented 10 months ago

Hey @junpeiz , if I understand well, the following lines are the reason for the bug:

# KO
# Compute vertical line
x = (landmarks[:, 9, 0] + landmarks[:, 13, 0]) / 2
y = (landmarks[:, 9, 1] + landmarks[:, 13, 1]) / 2

It WORKS if I remove them and replace x, y (faking it) with:

# OK
# Compute vertical line
x = tf.constant(np.random.rand(1, 1, 1), dtype=tf.float32)
y = tf.constant(np.random.rand(1, 1, 1), dtype=tf.float32)

But it FAILS if i try to use tf.gather to select specific indexes like this:

# KO
# Compute vertical line
xy = (tf.gather(landmarks, 9, axis=1) + tf.gather(landmarks, 13, axis=1)) / 2
x = tf.gather(xy, 0, axis=1)
y = tf.gather(xy, 1, axis=1)

or

# KO
# Compute vertical line
line_vec_indexes = [9, 13]
x = tf.math.reduce_mean(tf.gather(landmarks, line_vec_indexes, axis=1)[:, :, 0], axis=(1,))
y = tf.math.reduce_mean(tf.gather(landmarks, line_vec_indexes, axis=1)[:, :, 1], axis=(1,))

Am I misunderstanding your advice ? Can you please give me more details about the quick workaround I can do ? Thank you.