Different face detection and match results between browser and server

[dancherb]

Issue Description

Hi - we're using the library to take two photos of a user's face on the browser (after a face is detected) and to check for a match. Then, the results are sent to the server, which repeats the face detection and face match to confirm.

However - I'm experiencing different results on the two environments. Sometimes the server does not detect a face in the photos passed, or does not approve a match.

Steps to Reproduce

Browser:

    faceapi.matchDimensions(canvasRef.current, displaySize);

    const image = capturePhotoFromCamera({ videoRef })

    const detection = await faceapi.detectSingleFace(videoRef.current, new faceapi.SsdMobilenetv1Options())
    .withFaceLandmarks()
    .withFaceDescriptor();

const differenceScore = faceapi.euclideanDistance(faceA.descriptor, faceB.descriptor);

Server:

const detectFace = async ({ imageBuffer }) => {    
    const decodeT = faceapi.tf.node.decodeImage(imageBuffer, 3);
    const expandT = faceapi.tf.expandDims(decodeT, 0);

    const detection = await faceapi.detectSingleFace(expandT, new faceapi.SsdMobilenetv1Options())
    .withFaceLandmarks()
    .withFaceDescriptor();

    faceapi.tf.dispose([decodeT, expandT]); // dispose tensors to avoid memory leaks

    return detection
}

const differenceScore = faceapi.euclideanDistance(faceA.descriptor, faceB.descriptor);

Is there a reason this might be occurring? Thanks :)

**Environment

• Module version? 1.7.9
• Built-in demo or custom code? Custom
• Browser or NodeJS and version (e.g. NodeJS 14.15 or Chrome 89)? Node v16.13.1, latest version of Brave/Chrome
• OS and Hardware platform (e.g. Windows 10, Ubuntu Linux on x64, Android 10)? Windows 10
• Packager (if any) (e.g, webpack, rollup, parcel, esbuild, etc.)? Next.js

[vladmandic]

two quick questions:

• which backend are you using in browser? webgl or wasm?
• which version of @tensorflow/tfjs-node are you using on the server?
• how are you loading faceapi in the browser? next.js is notorious for trying to do everything using server-side-rendering.

"Sometimes the server does not detect a face in the photos passed, or does not approve a match."

can you elaborate on this? perhaps provide example of image that is processed on both sides and actual results you're getting? some difference is expected since precision of math operations in browser and in nodejs does vary by a tiny fraction, but it should not be that significant.

[dancherb]

Hey Vladimir, thanks for reaching out.

which backend are you using in browser? webgl or wasm?

I'm not completely sure what this is referring to - there isn't an explicit reference to either of these in the code. The sample I gave pretty much covers it - besides importing the package import * as faceapi from '@vladmandic/face-api';. I can see that @tensorflow/tfjs-backend-webgl does appear in our package-lock.json.

which version of @tensorflow/tfjs-node are you using on the server?

4.2.0

how are you loading faceapi in the browser? next.js is notorious for trying to do everything using server-side-rendering.

All of the faceapi logic occurs after the component is mounted, so it is all occurring on the client side.

can you elaborate on this? perhaps provide example of image that is processed on both sides and actual results you're getting? some difference is expected since precision of math operations in browser and in nodejs does vary by a tiny fraction, but it should not be that significant.

Yes, a sample could ge a good idea - I'll look into creating this.

But I found that when calculating the euclideanDistance, the client could say 0.66 but the server could say 0.64. If our threshold for a "pass" was 0.65, this meant inconsistent results. And yes, sometimes when we passed the exact same photo to the server it didn't detect a face at all.

[vladmandic]

Difference between 0.64 and 0.66 might be due to node VS browser math precision differences, but if you can provide examples I can take a look.

[vladmandic]

any updates?

[vladmandic]

closing due to inactivity, can be reopened if new information is provided.