Unable to integrate it with react native

[sedaplaksa]

tried to integrate the face api in react native but tfjs-image-recognition is in fs format. is there any way to resolve issue. Really need help as this is a school project . Thanks

[justadudewhohacks]

What is a fs format? Can describe the issue more precisely?

[sedaplaksa]

when i did an npm i face-api.js and import * as faceapi from 'face-api.js' into my page , it threw me an error fs module does not exist in module haste map

Did a check online and found the tjfs image recognition which is in fs format can only be work in web browser.

So im wondering is that anyway I can solve this error and make it run on mobile? I really need help on these issues . thanks.

[justadudewhohacks]

Did a check online and found the tjfs image recognition which is in fs format can only be work in web browser.

I am maintaing tfjs-image-recognition base, but I still don't know what "fs format" should be. A more detailed error message would be helpful.

[sedaplaksa]

Just a quick ask is it possible for faceapi to be used on react native currently?

[sedaplaksa]

I suspect this maybe where the error come from since react native cannot read 'fs' file format

[wulforr]

I too am facing the same problem in React-Native I have installed face-api.js and then imported it as import * as faceapi from 'face-api.js'; Then I get the following errors

 bundling failed: Error: Unable to resolve module `fs` from `node_modules/tfjs-image-recognition-base/build/commonjs/env/createFileSystem.js`: fs could not be found within the project.

If you are sure the module exists, try these steps:
 1. Clear watchman watches: watchman watch-del-all
 2. Delete node_modules: rm -rf node_modules and run yarn install
 3. Reset Metro's cache: yarn start --reset-cache
 4. Remove the cache: rm -rf /tmp/metro-*
    at ModuleResolver.resolveDependency (/root/React-native/faceapi/node_modules/metro/src/node-haste/DependencyGraph/ModuleResolution.js:186:15)
    at ResolutionRequest.resolveDependency (/root/React-native/faceapi/node_modules/metro/src/node-haste/DependencyGraph/ResolutionRequest.js:52:18)
    at DependencyGraph.resolveDependency (/root/React-native/faceapi/node_modules/metro/src/node-haste/DependencyGraph.js:282:16)
    at Object.resolve (/root/React-native/faceapi/node_modules/metro/src/lib/transformHelpers.js:267:42)
    at /root/React-native/faceapi/node_modules/metro/src/DeltaBundler/traverseDependencies.js:426:31
    at Array.map (<anonymous>)
    at resolveDependencies (/root/React-native/faceapi/node_modules/metro/src/DeltaBundler/traverseDependencies.js:423:18)
    at /root/React-native/faceapi/node_modules/metro/src/DeltaBundler/traverseDependencies.js:275:33
    at Generator.next (<anonymous>)
    at asyncGeneratorStep (/root/React-native/faceapi/node_modules/metro/src/DeltaBundler/traverseDependencies.js:87:24)

[lacson17]

Hi @sedaplaksa @wulforr,

It could very much be possible that tfjs-image-recognition-base is currently incompatible with React Native...

Node is only used by the packager to serve/compile your app bundle; in other words, React Native apps don't run in the node environment.

"fs" stands for file system, and in order to access it in React Native, you would have to use something like react-native-fs or rn-nodeify in order to bridge and talk to each platform's native APIs (the iOS/Android platforms, in this case). If you could find a way to integrate these into tfjs-image-recognition-base for your specific app use-case, then it should work.

@justadudewhohacks please correct me if I'm wrong in anything that I've stated above.

[msvargas]

I think its posible fix, you add 'fs empty' in webpack and create custom fetch in faceapi.env.monkeyPatch, but i dont know if this working in react-native

[sedaplaksa]

Hi @sedaplaksa @wulforr,

It could very much be possible that tfjs-image-recognition-base is currently incompatible with React Native...

Node is only used by the packager to serve/compile your app bundle; in other words, React Native apps don't run in the node environment.

"fs" stands for file system, and in order to access it in React Native, you would have to use something like react-native-fs or rn-nodeify in order to bridge and talk to each platform's native APIs (the iOS/Android platforms, in this case). If you could find a way to integrate these into tfjs-image-recognition-base for your specific app use-case, then it should work.

@justadudewhohacks please correct me if I'm wrong in anything that I've stated above.

i tried this method but it still did not work sadly :(

[sedaplaksa]

Hi does anyone have any other different solutions? Or is it just not compatible at the moment ?

[wulforr]

Well i removed face-api from react native and used it in node and sent my request from react native to node.

[sedaplaksa]

Do u mind sharing with me how u did it? Is there any latency during detection ?

[deividsoncs]

Hi guys, @wulforr can you give more details of you solution, im with the same problem, but i can turn it to server-side, i have to solve this issue on mobile app.

[artem-kushal]

Did anyone try integrate face api with this adapter?https://github.com/tensorflow/tfjs/tree/master/tfjs-react-native

[deividsoncs]

Did anyone try integrate face api with this adapter?https://github.com/tensorflow/tfjs/tree/master/tfjs-react-native

@artem-kushal may be one way to do this work, i didn't konw tfjs-react-native, i will try use it, please comment bellow if you got success...

[wulforr]

Hi guys, @wulforr can you give more details of you solution, im with the same problem, but i can turn it to server-side, i have to solve this issue on mobile app.

Sure @deividsoncs, I used face-api.js to find similarities between two images. I had images stored in the database and sent my image from mobile app to server. In server this image is compared to all the images in the database.

If you wanna know more or anything in specific do let me know.

[deividsoncs]

Hi guys, @wulforr can you give more details of you solution, im with the same problem, but i can turn it to server-side, i have to solve this issue on mobile app.

Sure @deividsoncs, I used face-api.js to find similarities between two images. I had images stored in the database and sent my image from mobile app to server. In server this image is compared to all the images in the database.

If you wanna know more or anything in specific do let me know.

Thanks @wulforr , but my problem is that a have to solve in a mobile app to send only the features extracted from the image captured in a mobile camera, because the image is taking so long time to uploado to the servers, im live at Brazil and the mobile internet connection is so slow in many places here. So, i wish do this at app, could you help on this?

[LauraPeraltaV85]

Hi guys, @wulforr can you give more details of you solution, im with the same problem, but i can turn it to server-side, i have to solve this issue on mobile app.

Sure @deividsoncs, I used face-api.js to find similarities between two images. I had images stored in the database and sent my image from mobile app to server. In server this image is compared to all the images in the database.

If you wanna know more or anything in specific do let me know.

Hi @wulforr, by any chance is your repo of this project public, if yes can you please tell me the name to check it out, I have been looking for examples of how to use face-api.js and I'm a little confused, thank you

[ibaam97]

I assume the library can be implemented in a react native webview

[CHOMNANP]

If anyone manage to make face-api work with react-native, please share your solution. I'm trying to figure out as well.

[yasahmed]

use https://www.npmjs.com/package/react-native-fs instead of fs

[sandys]

@yasahmed have you got face-api.js to work with react native ? can you post an example ?

[0xori]

as said before, you can use https://www.npmjs.com/package/react-native-fs instead of fs, but it's not enough. you'll need to install https://www.npmjs.com/package/@tensorflow/tfjs-react-native. use faceapi.setEnv to initialize, face-api is excpecting an Environment interface

{
  Canvas: typeof HTMLCanvasElement
  CanvasRenderingContext2D: typeof CanvasRenderingContext2D
  Image: typeof HTMLImageElement
  ImageData: typeof ImageData
  Video: typeof HTMLVideoElement
  createCanvasElement: () => HTMLCanvasElement
  createImageElement: () => HTMLImageElement
  fetch: (url: string, init?: RequestInit) => Promise<Response>
}

I was using react-native-canvas for the canvas elements here is a partial implementation

var RNFS = require('react-native-fs');
import Canvas, {
  Image,
  ImageData,
  CanvasRenderingContext2D,
} from 'react-native-canvas';
import * as faceapi from 'face-api';
import { cameraWithTensors } from '@tensorflow/tfjs-react-native';
import { Camera } from 'expo-camera';
const TensorCamera = cameraWithTensors(Camera);

const createCanvasElement = function () {
  if (Canvas) {
    return new Canvas();
  }
  throw new Error(
    'createCanvasElement - missing Canvas implementation for RN environment',
  );
};

const createImageElement = function () {
  if (Image) {
    return new Image();
  }
  throw new Error(
    'createImageElement - missing Image implementation for RN environment',
  );
};

const readFile = function (filePath: string) {
  try {
    return RNFS.readFile(filePath);
  } catch (e) {
    console.error('error reading file', e);
  }
};

const fileSystem = {
  readFile,
};

const _fetch =
  fetch ||
  function () {
    throw new Error(
      'fetch - missing fetch implementation for browser environment',
    );
  };

 let env = {
    Canvas: Canvas,
    CanvasRenderingContext2D: CanvasRenderingContext2D,
    Image: Image,
    ImageData: ImageData,
    Video: TensorCamera,
    createCanvasElement,
    createImageElement,
    fetch: _fetch,
    ...fileSystem,
  };

  faceapi.env.setEnv(env);

for loading the models i uploaded them to external storage (AWS S3) and used await faceapi.nets.tinyFaceDetector.loadFromUri('https://your-bucket-urls3.amazonaws.com/models');

and I used this example to get a proper input that I can pass to the faceDetector: https://github.com/tensorflow/tfjs/blob/master/tfjs-react-native/integration_rn59/components/webcam/realtime_demo.tsx

Everything haw worked without any errors but I couldn't get any results, also the detection process was very slow compared to web browser.

Good Luck, and please let us know if you have any breakthrough

[gravestar]

@0xori im trying to do step that you mention, but im still getting error, did u manage to finish that code ?

[samcladson]

I am facing the same issue. But somehow I managed to run the application without any errors up to the detection process following @0xori instruction except for the "expo-camera" I used "react-native-camera". But when I pass the camera ref as a parameter to the faceapi.detectSingleFace() function, I'm getting the following error; media.addEventListener("load", "onLoad") is not a function!

[TrungJamin]

Is there anybody having the solution for using face-api.js in RN ? I am trying to use some solutions above but it seems not to work TT

[vagnerpene]

Has anyone by chance managed to integrate face-api.js with RN? Any help will be welcome. Thanks in advance

[GuihMatos]

Hello, I've also been facing difficulties to use face-api.js with React Native (RN).

Here's my question: Is it possible to use POST and GET with face-api.js and React Native? For example: taking two photos with 'expo-camera' (RN) and making posts to face-api.js for the 'Face Recognition' model to determine if there is the same person in both images or not. Thanks for any help

[tinni95]

@GuihMatos that would be handy as well, I just don't think they have a REST api set up at the moment :(.

I would need to use this API for video chat detection, so I think making my own node.js server is not an acceptable solution because of latency. Please if anyone solves this could you please create a fork and share it? I will do the same if I find the solution first

Cheers :)

[jan-bogaerts]

So, I managed to get this library running with react-native, using react native tensorflow. This was a major, major pain to set up and get running. And, when run in debug mode, everything works fine: faces are detected, features are extracted in such a way that they can be compared to each other to find matching faces. However, when switching to release mode, the model breaks, that is to say: everything appears to run ok, but the face-detection module always comes back that it can't find any faces. upon further investigation, I found that the tensor values start to diverge during the calculations in the function mobileNetV1(x: tf.Tensor4D, params: MobileNetV1.Params) when this loop is executed: convPairParams.forEach((param, i). When i=3, calculations start to differ between debug and release mode. I don't yet know why exactly this is. I am currently thinking that it might have something to do with how the weights are loaded:

       const modelJson = require('../models/ssd_mobilenetv1/ssd_mobilenetv1_model-weights_manifest.json');
        let modelWeights1 = require('../models/ssd_mobilenetv1/ssd_mobilenetv1_model-shard1.bin');
        let modelWeights2 = require('../models/ssd_mobilenetv1/ssd_mobilenetv1_model-shard2.bin');

        let assetPath = Image.resolveAssetSource(modelWeights1);
        let response = await fetch(assetPath.uri, {}, { isBinary: true });
        const weights1 = await response.arrayBuffer();

        assetPath = Image.resolveAssetSource(modelWeights2);
        let response2 = await fetch(assetPath.uri, {}, { isBinary: true });
        const weights2 = await response2.arrayBuffer();

        const fetchWeightsFromDisk = (filePaths) => {
            console.log(filePaths);
            //console.log([...new Uint8Array(weights2.slice(0, 100))]);
            return Promise.resolve([weights1, weights2])
        }
        const loadWeights = tf.io.weightsLoaderFactory(fetchWeightsFromDisk);
        const weightMap = await loadWeights(modelJson, "");

maybe some byte mis-alignment? Does anybody have any idea?

So, I tried to make a fresh conversion from the old pb file, which worked almost, except for the 'NonMaxSuppression' operator. Does anybody here know how to modify the operators 'on the fly' after loading the model?

[vagnerpene]

Hi @jan-bogaerts I tried to integrate Face-api using react native tensorflow without success. Do you mind to share your implementation? Any help will be useful. Thanks

[jan-bogaerts]

@vagnerpene https://github.com/jan-bogaerts/face-api.js not much of a point though, it doesn't work in production mode, but does in debug mode. When the model gets executed on the device itself, there appears to be something going different.

[sametcl2]

I will use this library for face similarity and after reading the comments I decided to use webview. I wonder if the library works smoothly in webview?