Converted mediapipe palm detection coreml model cannot be deployed on IOS app

[TaoZappar]

Issue Type

Bug

OS

Mac OS

OS architecture

x86_64

Programming Language

Python

Framework

CoreML

Download URL for tflite file

https://github.com/google/mediapipe/blob/master/mediapipe/modules/palm_detection/palm_detection_full.tflite

Convert Script

docker run -it --rm \
  -v `pwd`:/home/user/workdir \
  ghcr.io/pinto0309/tflite2tensorflow:latest

tflite2tensorflow \
  --model_path palm_detection_full.tflite \
  --flatc_path ../flatc \
  --schema_path ../schema.fbs \
  --output_pb

tflite2tensorflow \
  --model_path palm_detection_full.tflite \
  --flatc_path ../flatc \
  --schema_path ../schema.fbs \
  --output_no_quant_float32_tflite \
  --output_dynamic_range_quant_tflite \
  --output_weight_quant_tflite \
  --output_float16_quant_tflite \
  --output_integer_quant_tflite \
  --string_formulas_for_normalization 'data / 255.0' \
  --output_tfjs \
  --output_coreml \
  --output_tftrt_float32 \
  --output_tftrt_float16 \
  --output_onnx \
  --onnx_opset 13 \
  --output_openvino_and_myriad

Description

when I deployed the converted mp palm detection coreml model on IOS app, I got error like:

2022-02-15 16:31:50.577312+0000 MLModelCamera[12602:4637828] [espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Generic error": [Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1

seems like the resizeBilinear op causes this problem. Is there any way to fix this issue? Thanks in advance.

The related code in tflite2tensorlfow.py is:

        elif op_type == 'RESIZE_BILINEAR':
            input_tensor = tensors[op['inputs'][0]]
            size_detail = interpreter._get_tensor_details(op['inputs'][1])
            size = interpreter.get_tensor(size_detail['index'])
            size_height = size[0]
            size_width  = size[1]

            options = op['builtin_options']
            align_corners = options['align_corners']
            half_pixel_centers = options['half_pixel_centers']

            def upsampling2d_bilinear(x, size_height, size_width, align_corners, half_pixel_centers):
                if optimizing_for_edgetpu_flg:
                    return tf.image.resize_bilinear(x, (size_height, size_width))
                else:
                    if optimizing_for_openvino_and_myriad:
                        if half_pixel_centers:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=False,
                                half_pixel_centers=half_pixel_centers
                            )
                        else:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=True,
                                half_pixel_centers=half_pixel_centers
                            )
                    else:
                        return tfv2.image.resize(
                            x,
                            [size_height, size_width],
                            method='bilinear'
                        )

            output_detail = interpreter._get_tensor_details(op['outputs'][0])
            lambda_name = get_op_name(output_detail['name']) + '_lambda'
            output_tensor_lambda = tf.keras.layers.Lambda(
                upsampling2d_bilinear,
                arguments={'size_height': size_height,
                            'size_width': size_width,
                            'align_corners': align_corners,
                            'half_pixel_centers': half_pixel_centers},
                name=lambda_name
            )(input_tensor)
            output_tensor = tf.identity(
                output_tensor_lambda,
                name=get_op_name(output_detail['name'])
            )
            tensors[output_detail['index']] = output_tensor

Relevant Log Output

2022-02-15 16:31:50.577312+0000 MLModelCamera[12602:4637828] [espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Generic error": [Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1

Source code for simple inference testing code

No response

[PINTO0309]

Sad to say, I don't own an IOS environment, so I can't test it. I'm not entirely sure if it will work, but could you please check if the CoreML model you downloaded from the following URL works? model_coreml_float32.mlmodel.zip

docker run -it --rm \
-v `pwd`:/home/user/workdir \
ghcr.io/pinto0309/tflite2tensorflow:latest

wget https://github.com/google/mediapipe/raw/master/mediapipe/modules/palm_detection/palm_detection_full.tflite

tflite2tensorflow \
--model_path palm_detection_full.tflite \
--flatc_path ../flatc \
--schema_path ../schema.fbs \
--output_pb

tflite2tensorflow \
--model_path palm_detection_full.tflite \
--flatc_path ../flatc \
--schema_path ../schema.fbs \
--output_coreml

[TaoZappar]

Thanks for the reply. Just test the model on the IOS, still get the same error.

2022-02-16 12:04:35.315685+0000 MLModelCamera[12740:4737111] [espresso] [Espresso::handle_ex_plan] exception=Espresso exception: "Generic error": [Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1
2022-02-16 12:04:35.315728+0000 MLModelCamera[12740:4737111] [coreml] Error computing NN outputs -1
2022-02-16 12:04:35.315755+0000 MLModelCamera[12740:4737111] [coreml] Failure in -executePlan:error:.
failed to perform

I checked the layer 240 of the graph, pretty sure the ResizeBilinear caused this problem, as it has also been mentioned in many onnx->coreml conversion, e.g. :

(https://github.com/onnx/onnx-coreml/issues/474) (https://github.com/apple/coremltools/issues/1105)

I also noticed that the new height and new width in resizebilinear layer are all 0. Is that something related?

Thanks

[PINTO0309]

I don't think the number 240 in the error message is directly related to the error. The error occurs in the operation named model_1/model/add_24/add.

[Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1

The shape of the tensor shown in the error message is obviously wrong.

Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1]

This is because there are no operations in this model that process in 5 dimensions.

I also noticed that the new height and new width in resizebilinear layer are all 0. Is that something related?

No, not at all. It is correct that new_height and new_width are set to zero, as per the specification. Since the second input is set to [24, 24], ResizeBilinear (upsample) will produce an output of size 24 in height and width.

The way I see it, it's a bug in CoreML.

[TaoZappar]

Thanks for the reply. I also think it's a bug in coremltool. However, I try to find a way to work around it. My initial thinking is to use upsample2d to replace image.resize method in ResizeBilinear Op. Currently, got the model deployed on iOS, but the size of the final output is incorrect. I will keep tracking and let u know once I sort it out.

Katsuya Hyodo @.***> 于 2022年2月16日周三 下午2:34写道：

I don't think the number 240 in the error message is directly related to the error. The error occurs in the operation named model_1/model/add_24/add.

[Exception from layer 240: model_1/model/add_24/add]Espresso exception: "Invalid blob shape": elementwise_kernel_cpu: Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1] status=-1

[image: image] https://user-images.githubusercontent.com/33194443/154284738-9a42f361-3645-4248-983c-df8bad275b36.png

The shape of the tensor shown in the error message is obviously wrong.

Cannot broadcast [12, 2, 1, 256, 1] and [12, 1, 1, 256, 1]

This is because there are no operations in this model that process in 5 dimensions.

I also noticed that the new height and new width in resizebilinear layer are all 0. Is that something related?

No, not at all. It is correct that new_height and new_width are set to zero, as per the specification. Since the second input is set to [24, 24], ResizeBilinear (upsample) will produce an output of size 24 in height and width. [image: image] https://user-images.githubusercontent.com/33194443/154285717-5a510b64-e6f6-4bf0-8fcd-240956d6df76.png

The way I see it, it's a bug in CoreML.

— Reply to this email directly, view it on GitHub https://github.com/PINTO0309/tflite2tensorflow/issues/27#issuecomment-1041554494, or unsubscribe https://github.com/notifications/unsubscribe-auth/AV7EQ3SUWDMYQY3PDEUKVZLU3OYV7ANCNFSM5ORKIUNQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

[TaoZappar]

Hi, Katsuya

I am glad to tell you I find a way to work around. I was inspired by this article https://machinethink.net/blog/coreml-upsampling/ (go to the Cheatsheet section) Instead of using tf.image.resize, I use tf.compat.v1.image.resize_bilinear. The hacked code I used for coreml specifically is as follows:

        elif op_type == 'RESIZE_BILINEAR':
            input_tensor = tensors[op['inputs'][0]]
            size_detail = interpreter._get_tensor_details(op['inputs'][1])
            size = interpreter.get_tensor(size_detail['index'])
            size_height = size[0]
            size_width = size[1]

            options = op['builtin_options']
            align_corners = options['align_corners']
            half_pixel_centers = options['half_pixel_centers']

            def upsampling2d_bilinear(x, size_height, size_width, align_corners, half_pixel_centers):
                if optimizing_for_edgetpu_flg:
                    return tf.image.resize_bilinear(x, (size_height, size_width))
                else:
                    if optimizing_for_openvino_and_myriad:
                        if half_pixel_centers:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=False,
                                half_pixel_centers=half_pixel_centers
                            )
                        else:
                            return tf.image.resize_bilinear(
                                x,
                                (size_height, size_width),
                                align_corners=True,
                                half_pixel_centers=half_pixel_centers
                            )
                    else:
                        y = tf.compat.v1.image.resize_bilinear(x, [size_height, size_width], align_corners=True) # slightly better way, which works on IOS
                        # y = tfv2.keras.layers.UpSampling2D()(x) #   no matching solution in COREML
                        # y = tfv2.image.resize(                  #   no matching solution in COREML
                        #     x,
                        #     [size_height, size_width],
                        #     method='bilinear'
                        # )
                        return y

[PINTO0309]

I see. I will add an option to enable CoreML optimization. Thank you for doing some meaningful research! :smile_cat:

[PINTO0309]

Commit: ab7d9625aa2abc9d006e03d27f6d7f534e83f4db Release: https://github.com/PINTO0309/tflite2tensorflow/releases/tag/v1.19.0