Can this work in react native webview

jk939393 commented 4 years ago

Hello, I saw through discussion there is no fully native support for face.api js through tensorflow, however would there be functionality through react webview

https://facebook.github.io/react-native/docs/webview.html

Thanks

justadudewhohacks commented 4 years ago

I do not see any reason, why this wouldn't work in a webview, I haven't tried it out yet though.

mayankagarwals commented 3 years ago

I don't think react-native-webview supports webgl yet so this might be slow. Although my react native knowledge is limited, so it'd be great if someone can reply with a workaround

manolo-battista commented 1 year ago

I can finally run face-api-js through the webview on ReactNative :)

I load a locally index.html with face-api detection code:

<html>
    <head>
        <style>
            img {
                z-index: 0;
            }
            #overlay, .overlay {
                position: absolute;
                top: 0;
                left: 0;
                max-width: 800px;
                z-index: 10;
            }
        </style>
    </head>
    <body>
        <div style="position:relative;background: red;width: 800px;height:800px">
            <canvas class="overlay" id="face_detection_canvas"></canvas>
            <img id="img"
             crossorigin="anonymous"
             alt="face"
             src="../../face-test.png"
             width="800"
             height="800"/>
        </div>
    </body>
    <script src="https://unpkg.dev/@vladmandic/face-api/dist/face-api.js"></script>
    <script>
        async function detect(){
            try{
                const canvas = document.getElementById("face_detection_canvas");
                const img = document.getElementById("img");
                await faceapi.loadSsdMobilenetv1Model('../../models/ssd_mobilenetv1')
                await faceapi.loadFaceLandmarkModel('../../models/face_landmark_68') // model to detect face landmark
                let faceDescriptions = await faceapi.detectAllFaces(img).withFaceLandmarks();
                faceapi.matchDimensions(canvas, img)
                const out = faceapi.createCanvasFromMedia(img);
                faceDescriptions = faceapi.resizeResults(faceDescriptions, img)
                faceapi.draw.drawDetections(canvas, faceDescriptions) //to draw box around detection
                faceapi.draw.drawFaceLandmarks(canvas, faceDescriptions) //to draw face landmarks
            } catch(e){
               console.log(e)
            }
        }

        window.onload = function(){
            detect()
        }
    </script>
</html>

I set the /models on the assets folder.

On React Native side I set a simple WebView.

import {View} from 'react-native';
import tw from 'twrnc';
import WebView from 'react-native-webview';

export default function FaceDetection() {
  return (
    <View style={tw`w-full h-full`}>
      <WebView
        source={require('./index.html')}
        javaScriptEnabled
        useWebKit
        setDomStorageEnabled
        setAppCacheEnabled
        setLoadsImagesAutomatically
        scalesPageToFit
        domStorageEnabled
        allowFileAccessFromFileURLs
        startInLoadingState
        originWhitelist={['*']}
        mixedContentMode="compatibility"
        onLoadEnd={e => {
          console.log('WebView', e.nativeEvent.url);
        }}
        onError={syntheticEvent => {
          const {nativeEvent} = syntheticEvent;
          console.warn('WebView error: ', nativeEvent);
        }}
        onMessage={event => console.log(event.nativeEvent.data)}
      />
    </View>
  );
}

simulator_screenshot_DCD77C39-F2E1-4BC0-A758-11F362418698

igorskiter commented 10 months ago

@manolo-battista how did you organize the folders? I'm having trouble importing files and loading models

manolo-battista commented 10 months ago

@manolo-battista how did you organize the folders? I'm having trouble importing files and loading models

I added index.html in the same folder of my FaceDetection.tsx file. The webview import it with:

  <WebView
        ...
      source={require('./index.html')}
/>

Than the index.html load the models with absolute path:

await faceapi.loadSsdMobilenetv1Model('../../models/ssd_mobilenetv1')
await faceapi.loadFaceLandmarkModel('../../models/face_landmark_68')

I can load correctly with Debug mode, the problem was when I release it cause the absolute path is not resolve correctly. I had no time to test it again, I don't know if you had the same problem.

justadudewhohacks / face-api.js

Can this work in react native webview #515