FaceLandmarker.createFromModelPath() is not working

[shivambhardwaj120]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

Mac Mini,Ventura 13, Xcode 14.1

MediaPipe Tasks SDK version

@mediapipe/tasks-vision@10.0.1, @latest

Task name (e.g. Image classification, Gesture recognition etc.)

FaceLandmarker

Programming Language and version (e.g. C++, Python, Java)

Typescript

Describe the actual behavior

I am working with react native app.

"FaceLandmarker.createFromModelPath(vision,
          "https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task`"
      );" gave following error:

[ReferenceError: Property 'document' doesn't exist]

Describe the expected behaviour

It should work without error.

Standalone code/steps you may have used to try to get what you need

import React from 'react';
import type {PropsWithChildren} from 'react';
import { useState, useRef, useEffect } from 'react';
import {
  SafeAreaView,
  ScrollView,
  StatusBar,
  StyleSheet,
  Text,
  useColorScheme,
  View,
} from 'react-native';

import {
  Colors,
  DebugInstructions,
  Header,
  LearnMoreLinks,
  ReloadInstructions,
} from 'react-native/Libraries/NewAppScreen';
import * as md from "@mediapipe/tasks-vision"
type SectionProps = PropsWithChildren<{
  title: string;
}>;

function Section({children, title}: SectionProps): JSX.Element {
  const isDarkMode = useColorScheme() === 'dark';
  return (
    <View style={styles.sectionContainer}>
      <Text
        style={[
          styles.sectionTitle,
          {
            color: isDarkMode ? Colors.white : Colors.black,
          },
        ]}>
        {title}
      </Text>
      <Text
        style={[
          styles.sectionDescription,
          {
            color: isDarkMode ? Colors.light : Colors.dark,
          },
        ]}>
        {children}
      </Text>
    </View>
  );
}

function App(): JSX.Element {
  const [isTfReady, setIsTfReady] = useState(false);
    const [result, setResult] = useState('');
    const image = useRef(null);
    const [mobilenetmodel, setmobilenetmodel] = useState(null)
    useEffect(() => {
      (async () => {
        console.warn('Done');

        await load1();

      })();

    }, []);

    const load1 = async () => {
      try {
        // Load mobilenet.
        const vision = await md.FilesetResolver.forVisionTasks(
          "node_modules/@mediapipe/tasks-vision/wasm/vision_wasm_internal.wasm"
          //"https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision/wasm"
      );
      // const vision = await md.FilesetResolver.forVisionTasks();
      const faceLandmarker = await md.FaceLandmarker.createFromModelPath(vision,
          "https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task`"
      );

      } catch (err) {
        console.log(err);
      }

    };
  const isDarkMode = useColorScheme() === 'dark';

  const backgroundStyle = {
    backgroundColor: isDarkMode ? Colors.darker : Colors.lighter,
  };

  return (
    <SafeAreaView style={backgroundStyle}>
      <StatusBar
        barStyle={isDarkMode ? 'light-content' : 'dark-content'}
        backgroundColor={backgroundStyle.backgroundColor}
      />
      <ScrollView
        contentInsetAdjustmentBehavior="automatic"
        style={backgroundStyle}>
        <Header />
        <View
          style={{
            backgroundColor: isDarkMode ? Colors.black : Colors.white,
          }}>
          <Section title="Step One">
            Edit <Text style={styles.highlight}>App.tsx</Text> to change this
            screen and then come back to see your edits.
          </Section>
          <Section title="See Your Changes">
            <ReloadInstructions />
          </Section>
          <Section title="Debug">
            <DebugInstructions />
          </Section>
          <Section title="Learn More">
            Read the docs to discover what to do next:
          </Section>
          <LearnMoreLinks />
        </View>
      </ScrollView>
    </SafeAreaView>
  );
}

const styles = StyleSheet.create({
  sectionContainer: {
    marginTop: 32,
    paddingHorizontal: 24,
  },
  sectionTitle: {
    fontSize: 24,
    fontWeight: '600',
  },
  sectionDescription: {
    marginTop: 8,
    fontSize: 18,
    fontWeight: '400',
  },
  highlight: {
    fontWeight: '700',
  },
});

export default App;

Other info / Complete Logs

[ReferenceError: Property 'document' doesn't exist]

[schmidt-sebastian]

By default, FaceLandmarker should run in a browser context. It needs access to a Canvas for WebGL rendering. It looks like a) it cannot create an OffscreenCanvas and b) the fallback to create a canvas via document.createElement is not working. You should be able to work around this by setting the canvas property yourself: https://github.com/google/mediapipe/blob/master/mediapipe/tasks/web/vision/core/vision_task_options.d.ts#L34

[no-made]

Thank you for the answer. I have also tried giving a canvas, but maybe it was too big for the view. I have to try with a smaller one. Now that I'm writing I forgot to mention that I'm working on a nodejs project and that the script I wrote in the question is called in my html page as a module js script.

[schmidt-sebastian]

We currently don't support running under Node. You can develop in Node, but the task must be executed in a browser. How the JS is bundled should not impact how MediaPipe runs, as long as the browser can load the JavaScript code.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[shivambhardwaj120]

Hi @schmidt-sebastian thanks for the reply, I tried to set the canvas property and provided the offScreenCanvas. But still getting the same error.

const offscreen = Skia.Surface.MakeOffscreen(100,100); const faceLandmarker=await md.FaceLandmarker.createFromOptions(vision, { baseOptions: { modelAssetPath: https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/1/face_landmarker.task, delegate: "GPU" }, outputFaceBlendshapes: true, numFaces: 1, canvas: offscreen }); if I couldn't set it properly please tell me how to do it, because I couldn't find canvas property in any example.

[schmidt-sebastian]

This should be fixed with https://github.com/google/mediapipe/commit/990bfd2e3eaf0f0c977a7af7330ac4ad54f768cb. Please wait for a new release. You will still need to provide an OffscreenCanvas.