Error vision_wasm during ImageSegmenter initialization

[NikKotochigov]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

IOS 13.2.1

MediaPipe Tasks SDK version

0.10.12

Task name (e.g. Image classification, Gesture recognition etc.)

Image segmentation

Programming Language and version (e.g. C++, Python, Java)

Java Script

Describe the actual behavior

ImageSegmenter initialized

Describe the expected behaviour

error: opengl error checking is disabled vision_wasm_internal.js 9

Standalone code/steps you may have used to try to get what you need

import {
  FilesetResolver, ImageSegmenter, MPMask
} from '@mediapipe/tasks-vision';
import {
  IBackgroundSettings,
  IDrawOptions,
  VIRTUAL_BACKGROUND_TYPE,
} from 'apps/conference/shared';
import {
  BLUR_VALUE,
  FRAME_RATE,
  MEDIAPIPE_PATH, MODEL_ASSET_PATH,
  SEGMENTER_HEIGHT,
  SEGMENTER_WIDTH
} from 'apps/conference/shared/constants/background';
import {
  createCopyTextureToCanvas, createVirtualImageBitmap
} from './helpers';

let imageSegmenter: ImageSegmenter;
let toImageBitmap: (mask: MPMask) => Promise<ImageBitmap>;
let virtualImageBitmap: ImageBitmap | null = null;
const backgroundSource = '';

function checkFeaturesAvailability() {
  const isWebAssemblySupported = typeof WebAssembly === "object" &&
                                 typeof WebAssembly.instantiate === "function";

  const isMediaStreamTrackSupported = typeof MediaStreamTrack === "function";

  return {
    isWebAssemblySupported,
    isMediaStreamTrackSupported
  };
}

async function fetchModelAsset(): Promise<Uint8Array> {
  try {
    const response = await fetch(MODEL_ASSET_PATH);
    if (!response.ok) {
      throw new Error(`Failed to fetch model asset. Status: ${response.status} ${response.statusText}`);
    }
    const arrayBuffer = await response.arrayBuffer();
    return new Uint8Array(arrayBuffer);
  } catch (error) {
    console.log("Error fetching model asset:", error);
  }
}

async function initializeImageSegmenter() {
  if (!imageSegmenter) {
    const canvas = document.createElement("canvas");
    canvas.width = SEGMENTER_WIDTH;
    canvas.height = SEGMENTER_HEIGHT;

    const wasmFileset = await FilesetResolver.forVisionTasks(MEDIAPIPE_PATH);

    const start =  performance.now();
    // const modelAssetBuffer = await fetchModelAsset();

    imageSegmenter = await ImageSegmenter.createFromOptions(wasmFileset, {
      baseOptions: {
        modelAssetPath: MODEL_ASSET_PATH,
        delegate: "GPU"
      },
      canvas,
      runningMode: 'IMAGE'
    });

    const finish =  performance.now();
    console.log({ imageSegmenterInit: finish-start });

    toImageBitmap = createCopyTextureToCanvas(canvas);
  }
}

async function drawSegmentationResult(options: IDrawOptions): Promise<void> {
  const {
    maskImage,
    streamImage,
    canvas,
    backgroundType,
    virtualImageBitmap
  } = options;

  const {
    width, height
  } = canvas;

  const canvasCxt = canvas.getContext('2d');

  canvasCxt.save();
  canvasCxt.fillStyle = 'black';
  canvasCxt.clearRect(0, 0, width, height);

  canvasCxt.drawImage(maskImage, 0, 0, width, height);
  canvasCxt.restore();

  canvasCxt.save();
  canvasCxt.globalCompositeOperation = 'source-out';
  if (backgroundType === VIRTUAL_BACKGROUND_TYPE.IMAGE) {
    canvasCxt.drawImage(virtualImageBitmap, 0, 0, width, height);
  } else {
    canvasCxt.filter = `blur(${BLUR_VALUE}px)`;
    canvasCxt.drawImage(streamImage, 0, 0);
  }
  canvasCxt.restore();

  canvasCxt.save();
  canvasCxt.globalCompositeOperation = 'destination-atop';
  canvasCxt.drawImage(streamImage, 0, 0, width, height);
  canvasCxt.restore();

  maskImage.close();
  streamImage.close();
}

export async function startEffect(stream: MediaStream, backgroundSettings: IBackgroundSettings): Promise<MediaStream> {
  const {
    isWebAssemblySupported,
    isMediaStreamTrackSupported
  } = checkFeaturesAvailability();

  if (!isWebAssemblySupported) {
    throw new Error('WebAssembly not supported!');
  }

  if (!isMediaStreamTrackSupported) {
    throw new Error('MediaStreamTrack not supported!');
  }

  const videoTrack = stream.getVideoTracks()[0];
  await videoTrack.applyConstraints({ frameRate: FRAME_RATE });

  const {
    width, height
  } = videoTrack.getSettings();
  const {
    backgroundType, virtualSource
  } = backgroundSettings;

  console.log({ imageSegmenter });
  await initializeImageSegmenter();

  if (backgroundType === VIRTUAL_BACKGROUND_TYPE.IMAGE && (backgroundSource !== virtualSource || !virtualImageBitmap)) {
    virtualImageBitmap = await createVirtualImageBitmap(virtualSource);
  }
  // const finish1 = performance.now();
  // console.log({ virtualImageBitmap: finish1-start1 });

  const canvas = document.createElement("canvas");
  canvas.width = width;
  canvas.height = height;

  const processor = new MediaStreamTrackProcessor({ track: videoTrack });
  const generator = new MediaStreamTrackGenerator({ kind: 'video' });

  const readableStream = processor.readable;
  const writableStream = generator.writable;

  const transformer = new TransformStream<VideoFrame, VideoFrame>({
    async transform(videoFrame: VideoFrame, controller: TransformStreamDefaultController<VideoFrame>) {
      try {
        const streamImage = await createImageBitmap(videoFrame);

        const result = imageSegmenter.segment(streamImage);
        const maskImage = await toImageBitmap(result.confidenceMasks[0]);
        result.close();

        const drawOptions = {
          maskImage,
          streamImage,
          canvas,
          backgroundType,
          virtualImageBitmap
        }

        await drawSegmentationResult(drawOptions);

        const { timestamp } = videoFrame;
        const newFrame = new VideoFrame(canvas, { timestamp });
        videoFrame.close();
        controller.enqueue(newFrame);

      } catch (error) {
        videoFrame.close();
      }
    }
  })

  readableStream.pipeThrough(transformer).pipeTo(writableStream).catch((error: Error) => {
    readableStream.cancel();
    videoTrack.stop();
    generator.stop();
    console.error('Error pipe video frame:', error);
  });

  const processedStream = new MediaStream();
  processedStream.addTrack(generator);

  return processedStream;
}

Other info / Complete Logs

I think the error occurs here: const wasmFileset = await FilesetResolver.forVisionTasks(MEDIAPIPE_PATH);,

MEDIAPIPE_PATH is correct

[NikKotochigov]

[edamlmmv]

My implementation may be wrong but on my side I had to re-init the imageSegmenter every time I re-applied the effect on my canvas. I think I'm properly cleaning my canvas but it still crashes. Maybe try if it resolve your issue

That's a bummer because imageSegmenter takes awhile to construct. However, the wasmFileSet can be re-used.

[kuaashish]

Hi @NikKotochigov,

Could you please let us know if you are making any customizations to the code? Alternatively, can you provide the reference from our documentation that you are following, available here?

Thank you!!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No