faceDescriptors of different faces are exactly the same? What am I doing wrong?

[fakob]

Hi I am running faceapi.detectSingleFace(image).withFaceLandmarks().withAgeAndGender().withFaceDescriptor(); of different images/faces and most not all of them have the exact same faceDescriptor values. This seems wrong. Looking at their detection object, they clearly are different. What am I doing wrong?

Here is my code where I try to find unique faces.

export const detectFace = async (image, frameNumber, detectionArray, uniqueFaceArray) => {
  // detect expression
  const face = await faceapi.detectSingleFace(image).withFaceLandmarks().withAgeAndGender().withFaceDescriptor();

  console.log(frameNumber);
  // check if a face was detected
  if (face !== undefined) {
    const { age, gender, descriptor, detection } = face;
    const { relativeBox, score } = detection;
    const size = Math.round(relativeBox.height * 100);
    const scoreInPercent = Math.round(score * 100);

    // console.log(face);
    if (size < FACE_SIZE_THRESHOLD || scoreInPercent < FACE_DETECTION_CONFIDENCE_SCORE) {
      console.log('detected face below size or confidence threshold!');
      return undefined;
    }

    // create full copy of array to be pushed later
    const copyOfDescriptor = descriptor.slice();

    console.log(detection);
    console.log(uniqueFaceArray);
    // console.log(copyOfDescriptor);

    // initialise the faceId
    let faceId = 0;

    // check for uniqueness
    // if the uniqueFaceArray is empty just push the current descriptor
    // else compare the current descriptor to the ones in the uniqueFaceArray
    const uniqueFaceArrayLength = uniqueFaceArray.length;
    if (uniqueFaceArrayLength === 0) {
      uniqueFaceArray.push(copyOfDescriptor);
    } else {
      // compare descriptor value with all values in the array
      for (let i = 0; i < uniqueFaceArrayLength; i += 1) {
        const dist = faceapi.euclideanDistance(copyOfDescriptor, uniqueFaceArray[i]);
        console.log(`${faceId}, ${frameNumber}`);
        console.log(dist);
        // if no match was found add the current descriptor to the array marking a unique face
        if (dist < FACE_UNIQUENESS_THRESHOLD) {
          // console.log(`face matches face ${i}`);
          faceId = i;
          break;
        } else if (i === uniqueFaceArrayLength - 1) {
          console.log('face is unique');
          uniqueFaceArray.push(copyOfDescriptor);
          faceId = uniqueFaceArrayLength;
        }
      }
    }

    // console.log(`frameNumber: ${frameNumber}, Score: ${score}, Size: ${size}, Gender: ${gender}, Age: ${age}`);
    detectionArray.push({
      faceId,
      frameNumber,
      score: scoreInPercent,
      size,
      gender,
      age: Math.round(age),
    })
    return detection;
  }
  console.log('no face detected!');
  return undefined;
}

[justadudewhohacks]

exact same faceDescriptor value

This is indeed very suspicious, can you post an example image where this issue occurs?

The only two reasons I can think of is, that somethings wrong with your code, or theres something going wrong in the tfjs backend due to your system environment.

Could you give some information about your environment? (browser or nodejs, which browser / nodejs version, your OS and which backend you are using, cpu or webgl)

[fakob]

Thanks for looking into this.

Latest findings The issue with the same faceDescriptors only occurs with larger image files and only after a certain number of images. With my MoviePrint app I am using e.g. 20 images of a movie and feed it into face-api.js. When I feed in low resolution images (320x240), everything works as expected. When I do this with the same full image size (1920x1080), after the 3rd detected face it keeps throwing out the same faceDescriptor.

Does that help?

Setup and 2 example images: OS: macOS 10.14.6 Mojave Node version: v13.7.0 Electron: 4.2.10 Chromium: 69.0.3497.128 face-api.js: 0.22.0

[fakob]

You are probably right that it has something to do with my setup. I will investigate further.

[fakob]

Somehow I believe that the problem is with the image input as my boxes are already way off. I had not noticed that before.

My app is using opencv4nodejs to get the image data. I have tried 3 ways of getting my input into faceapi.detectAllFaces(input).

The first approach via an actual canvas brought the result seen in the image above:

faceapi.env.monkeyPatch({
  Canvas: HTMLCanvasElement,
  Image: HTMLImageElement,
  ImageData,
  Video: HTMLVideoElement,
  createCanvasElement: () => document.createElement('canvas'),
  createImageElement: () => document.createElement('img')
});

const mat = vid.read();

const input = document.getElementById('myCanvas');
input.height = mat.rows;
input.width = mat.cols;

const imgData = new ImageData(
new Uint8ClampedArray(matRescaled.getData()),
matRescaled.cols,
matRescaled.rows
);

const ctx = input.getContext('2d');
ctx.putImageData(imgData, 0, 0);

const detections = await faceapi.detectAllFaces(input);

Then I tried the other suggested method using node-canvas, but I don't know how to get from the opencv Mat to the image format needed. They both end in an error. I tried this.

const { Canvas, Image, ImageData } = canvas;
faceapi.env.monkeyPatch({ Canvas, Image, ImageData });

const mat = vid.read();

const input = new ImageData(
  new Uint8ClampedArray(mat.getData()),
  mat.cols,
  mat.rows
);

const detections = await faceapi.detectAllFaces(input);

and that approach

const { Canvas, Image, ImageData } = canvas;
faceapi.env.monkeyPatch({ Canvas, Image });

const mat = vid.read();

const input = new Image();

input.width = matRescaled.cols;
input.height = matRescaled.rows;
input.src = matRescaled.getData();

const detections = await faceapi.detectAllFaces(input);

What am I doing wrong to get from opencv Mat to the needed input format?

[fakob]

Forgot to post the error I am getting on the last approach and noticed that there is already an issue on that - https://github.com/justadudewhohacks/face-api.js/issues/194

[fakob]

Finally I made it work and it seems that reading the images properly also has solved the originally stated issue. I was unfortunately not successful with node-canvas, but made it work with tfjs-node. Thanks to @whyboris for the tip!

import * as tf from '@tensorflow/tfjs-node';

const input = tf.node.decodeJpeg(jpgImage);