"Batch size mismatch" Error when Using Selfie Multiclass Model with GPU Delegate

[soonhyo]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

Linux Ubuntu 20.04

MediaPipe Tasks SDK version

0.10.9

Task name (e.g. Image classification, Gesture recognition etc.)

selfie multicalss segemetation

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

I am encountering a "Batch size mismatch" error when attempting to use the Selfie Multiclass model with GPU Delegate in MediaPipe. The error message is as follows:WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1705492784.934668 18558 gl_context_egl.cc:85] Successfully initialized EGL. Major : 1 Minor: 5 I0000 00:00:1705492785.005127 18620 gl_context.cc:344] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 525.147.05), renderer: NVIDIA RTX A2000 Laptop GPU/PCIe/SSE2 INFO: Created TensorFlow Lite delegate for GPU. ERROR: TfLiteGpuDelegate Prepare: Batch size mismatch, expected 1 but got 8 ERROR: Node number 175 (TfLiteGpuDelegate) failed to prepare. ERROR: Restored original execution plan after delegate application failure. E0000 00:00:1705492785.049712 18558 calculator_graph.cc:876] INTERNAL: CalculatorGraph::Run() failed: Calculator::Open() for node "mediapipe_tasks_vision_image_segmenter_imagesegmentergraphmediapipe_tasks_core_inferencesubgraphinferencecalculatormediapipe_tasks_vision_image_segmenter_imagesegmentergraphmediapipe_tasks_core_inferencesubgraphInferenceCalculator" failed: ; RET_CHECK failure (mediapipe/calculators/tensor/inference_calculatorgl.cc:194) (interpreter->ModifyGraphWithDelegate(delegate_.get()))==(kTfLiteOk) Traceback (most recent call last): File "mp_head.py", line 156, in ros_app = RosApp() File "mp_head.py", line 106, in init super(RosApp, self).init() File "mp_head.py", line 60, in init self.segmenter = ImageSegmenter.create_from_options(self.options) File "/home/s-kim/.local/lib/python3.8/site-packages/mediapipe/tasks/python/vision/image_segmenter.py", line 268, in create_from_options return cls( File "/home/s-kim/.local/lib/python3.8/site-packages/mediapipe/tasks/python/vision/image_segmenter.py", line 145, in init super(ImageSegmenter, self).init( File "/home/s-kim/.local/lib/python3.8/site-packages/mediapipe/tasks/python/vision/core/base_vision_task_api.py", line 70, in init self._runner = _TaskRunner.create(graph_config, packet_callback) RuntimeError: CalculatorGraph::Run() failed: Calculator::Open() for node "mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraphinferencecalculatormediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraphInferenceCalculator" failed: ; RET_CHECK failure (mediapipe/calculators/tensor/inference_calculatorgl.cc:194) (interpreter->ModifyGraphWithDelegate(delegate_.get()))==(kTfLiteOk)

Describe the expected behaviour

I expected that when running the Selfie Multiclass model with GPU Delegate, the batch size would match the expected value, and no error would occur.

Standalone code/steps you may have used to try to get what you need

This is my code:
import cv2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from mediapipe.framework.formats import landmark_pb2
from mediapipe import solutions
import numpy as np
import time
from typing import List

BG_COLOR = (192, 192, 192)  # gray
BLACK_COLOR = (0, 0, 0)  # black
MASK_COLOR = (255, 255, 255)  # white
BODY_COLOR = (0, 255, 0)  # green
FACE_COLOR = (255, 0, 0)  # red
CLOTHES_COLOR = (255, 0, 255)  # purple

BaseOptions = mp.tasks.BaseOptions
ImageSegmenter = mp.tasks.vision.ImageSegmenter
ImageSegmenterOptions = mp.tasks.vision.ImageSegmenterOptions
VisionRunningMode = mp.tasks.vision.RunningMode

class App:
    def __init__(self):
        self.output_image = None

        self.base_options = python.BaseOptions(model_asset_path='selfie_multiclass_256x256.tflite',
                                               delegate=python.BaseOptions.Delegate.GPU)

        self.options = ImageSegmenterOptions(base_options=self.base_options,
                                             running_mode=VisionRunningMode.LIVE_STREAM,
                                             output_category_mask=True,
                                             output_confidence_masks=False,
                                             result_callback=self.mp_callback)

        self.segmenter = ImageSegmenter.create_from_options(self.options)

        self.latest_time_ms = 0

    def update(self, frame):
        t_ms = int(time.time() * 1000)
        if t_ms <= self.latest_time_ms:
            print("no update")
            return

        mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=frame)
        self.segmenter.segment_async(mp_image, t_ms)
        self.latest_time_ms = t_ms

    def mp_callback(self, segmentation_result: List[mp.Image], rgb_image: mp.Image, timestamp_ms: int):
        category_mask = segmentation_result.category_mask

        image_data = rgb_image.numpy_view()
        fg_image = np.zeros(image_data.shape[:2], dtype=np.uint8)
        fg_image[:] = MASK_COLOR[0]
        bg_image = np.zeros(image_data.shape[:2], dtype=np.uint8)
        bg_image[:] = BLACK_COLOR[0]

        condition1 = category_mask.numpy_view() == 1  # hair

        if np.sum(condition1) == 0:
            self.output_image = bg_image
        else:
            self.output_image = np.where(condition1, fg_image, bg_image)

if __name__ == "__main__":
    app = App()
    cap = cv2.VideoCapture(0)

    while True:
        ret, frame = cap.read()
        if ret:
            frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
            app.update(frame)

            if app.output_image is not None:
                cv2.imshow("Segmentation", app.output_image)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

            time.sleep(0.1)

    cap.release()
    cv2.destroyAllWindows()

Other info / Complete Logs

No response

[kuaashish]

Hi @soonhyo ,

It looks like you are trying to modify our Task API code using the Selfie Multiclass tflite model. We haven't tested this yet, we will be unable to offer support in this case. If you are following our documentation from here, please let us know the specific guide or instructions you are using so we can better understand and address the issue.

Thank you

[soonhyo]

Thank you for your reply.

I have referred to the live stream example for Python in the MediaPipe Image Segmentation documentation. Based on the discussion in this #4984, I added delegate=python.BaseOptions.Delegate.GPU to the code. There are no significant other modifications made to the example code.

It's worth noting that the same code worked with GPU support for the hair_segmenter.tflite model. Additionally, when I converted the selfie_multiclass_256x256.tflite model to an ONNX model, it operated successfully with GPU support.

[schmidt-sebastian]

Are you able to use the model in MediaPipe Studio? https://mediapipe-studio.webapps.google.com/studio/demo/image_segmenter

You can select the same model here (multi-class segmentation) and GPU as the delegate.

[soonhyo]

Thank you for your reply, @schmidt-sebastian. Yes, it worked normally in MediaPipe Studio, but it has not worked in the Python code yet.

[lambiengcode]

Same on Android 11T (21081111RG) works well with CPU delegate but catch "Batch size mismatch" error when modify to GPU. I use this device and use mediapipe studio demo, when selecting Multi-class Selfie Segmenter 256 and GPU it seems to have the same error, I think the problem is the GPU model on this device but when testing other models like Selfie Segmenter, Hair Segmenter and Deeplab V3 with the GPU delegate still working as expected

Log Details:

 E  Image segmenter failed to load model with error: invalid argument: CalculatorGraph::Run() failed in Run: 
                                                                                                    Calculator::Open() for node "mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_image_segmenter_imagesegmentergraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator" failed: Batch size mismatch, expected 1 but got 8