Browser Crashes When Loading MediaPipe Models After OpenCV.js Initialization

[NandanModtech]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Windows 11

MediaPipe Tasks SDK version

mediapipe/tasks-vision@0.10.14

Task name (e.g. Image classification, Gesture recognition etc.)

Face Landmarker & Hand Landmarker

Programming Language and version (e.g. C++, Python, Java)

Javascript

Describe the actual behavior

The browser crashes immediately when attempting to load mediapipe task vision ( face landmarker ) after loading hand landmarker & opencv.js together at initial.

Describe the expected behaviour

The MediaPipe Vision task models should load successfully without causing the browser to crash, allowing both OpenCV.js and MediaPipe to operate simultaneously.

Standalone code/steps you may have used to try to get what you need

import { FilesetResolver } from '@mediapipe/tasks-vision';

export let handLandmarker, faceLandmarker;
let initializedCvOnce = false;

/**
 * Initializes the Hand Landmarker
 * @param {Object} vision - Vision tasks resolver
 * @returns {Promise<Object>} - HandLandmarker instance
 */
const initializeHandLandMarker = async (mode) => {
  try {
    const { default: handLandMarkTask } = await import('./hand_landmarker.task');
    const { HandLandmarker } = await import('@mediapipe/tasks-vision');
    const vision = await FilesetResolver.forVisionTasks('https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm');
    handLandmarker = await HandLandmarker.createFromOptions(vision, {
      baseOptions: {
        modelAssetPath: handLandMarkTask,
        delegate: 'GPU',
      },
      numHands: 1,
      runningMode: mode,
      minHandDetectionConfidence: 0.7,
      minTrackingConfidence: 0.7,
    });

    return handLandmarker;
  } catch (error) {
    throw error;
  }
};

/**
 * Initializes the Face Landmarker
 * @param {Object} vision - Vision tasks resolver
 * @returns {Promise<Object>} - FaceLandmarker instance
 */
const initializeFaceLandMarker = async (mode) => {
  try {
    const { default: faceLandMarkTask } = await import('./face_landmarker.task');
    const { FaceLandmarker } = await import('@mediapipe/tasks-vision');
    const vision = await FilesetResolver.forVisionTasks('https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm');
    faceLandmarker = await FaceLandmarker.createFromOptions(vision, {
      baseOptions: {
        modelAssetPath: faceLandMarkTask,
        delegate: 'GPU',
      },
      numFaces: 1,
      minTrackingConfidence: 0.8,
      minFaceDetectionConfidence: 0.8,
      runningMode: mode,
      outputFacialTransformationMatrixes: true,
    });
    return faceLandmarker;
  } catch (error) {
    throw error;
  }
};

function loadOpenCv() {
  const scriptElement = document.createElement('script');
  scriptElement.src = 'assets/openCv/opencv.js';
  scriptElement.type = 'text/javascript';
  document.body.appendChild(scriptElement);
  let interval = setInterval(() => {
    if (window.cv) {
      clearInterval(interval);
      window['setUpCv']();
    }
  }, 500);
}

export const loadModels = async (detectionModel, isHandChoosen, mode) => {
  try {
    if (isHandChoosen) {
      detectionModel.handModel = await initializeHandLandMarker(mode);
    } else {
      detectionModel.faceModel = await initializeFaceLandMarker(mode);
    }

    if (!initializedCvOnce) {
      initializedCvOnce = true;
      loadOpenCv();
    }
  } catch (error) {
    console.log(error);
  }
};

Other info / Complete Logs

No response

[kuaashish]

Hi @NandanModtech,

Unfortunately, we cannot provide support for this issue. We recommend migrating your example to a Node environment for development and using a JavaScript bundler to compile your sources. This approach will consolidate your multiple script imports into a single import using JavaScript modules, reducing potential conflicts between the two libraries.

Additionally, consider using the package available at @mediapipe/tasks-vision.

Thank you!!

[NandanModtech]

I am using React. Should I use a package manager for OpenCV?

[kuaashish]

Hi @NandanModtech,

OpenCV compatibility has not been tested with our Task API. Therefore, we recommend using a Node environment for development to minimize conflicts between the two libraries. Based on past reported issues, we cannot provide support for this configuration, and it is not on our roadmap.

Thank you!!

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No