Load image take too much time

[danies8]

Hi, I used this face-api.js (https://github.com/justadudewhohacks/face-api.js) and I have to load 10000 images when the node server is loaded for future comparison operations. This the function that I load when server is loaded, it take 0.5 second to one picture, how can I improve it ?

**// fetch first image of each class and compute their descriptors
async function createBbtFaceMatcher() {

  await faceDetectionNet.loadFromDisk(path.join(__dirname, '../weights'))
  await faceapi.nets.faceLandmark68Net.loadFromDisk(path.join(__dirname, '../weights'))
  await faceapi.nets.faceRecognitionNet.loadFromDisk(path.join(__dirname, '../weights'))

  const labeledFaceDescriptors = await Promise.all(classes.map(
    async className => {
      let descriptors: any = [];
      let uri = getFaceImageUri(className, 1);
      const img = await canvas.loadImage(uri);
      if (img) {
        let descriptor = await faceapi.computeFaceDescriptor(img);
        if (descriptor) {
          descriptors.push(descriptor);
        }
      }
      return new faceapi.LabeledFaceDescriptors(
        className,
        descriptors
      )
    }
  )) 

return new faceapi.FaceMatcher(labeledFaceDescriptors) }**

[vladmandic]

you havent said which backend you're using - since it's nodejs solution, i'm assuming tfjs-node?

anyhow, javascript is not a platform to run compute intensive operations concurrently - and you're triggering all faceapi detections inside a promise - don't do that, run simple loop instead

also, note that this original version of face-api is using hard-coded embedded version of tfjs 1.7 which actually uses tensorflow.so version 1.15 for execution in tfjs-node backend which is quite old

you may want to try newer port of face-api that uses tfjs 3.9 which relies on tensorflow.so 2.6 which implements more accelerated functions

[vladmandic]

switching to loop doesn't make it faster - after all, it's the same code. it does make it use far less memory and less chances of race scenarios leading to crashes while still running with the same performance

[danies8]

@vladmandic Thanks for answer. How can I speed the load of 100000 images in few seconds instead of half hour now?

[danies8]

10,000

[vladmandic]

i just checked using your code (modified to work - it would help if you post fully working code when asking question) on my home server with i5-6500TE, so very low power and it's ~75ms per image, far from 0.5sec you're seeing. what's your hardware and what's the backend you're using (console.log(tf.getBackend()))?

const fs = require('fs');
const canvas = require('canvas');
const faceapi = require('./dist/face-api.node.js');

async function main() {
  faceapi.env.monkeyPatch({ Canvas: canvas.Canvas, Image: canvas.Image, ImageData: canvas.ImageData });
  const faceDetectionNet = new faceapi.FaceDetectionNet();
  await faceDetectionNet.loadFromDisk('model');
  await faceapi.nets.faceLandmark68Net.loadFromDisk('model');
  await faceapi.nets.faceRecognitionNet.loadFromDisk('model');
  const dir = fs.readdirSync('../human/samples/people');
  const descriptors = [];
  const t0 = performance.now();
  for (const f of dir) {
    const img = await canvas.loadImage(`../human/samples/people/${f}`);
    const c = canvas.createCanvas(img.width, img.height);
    const ctx = c.getContext('2d');
    ctx.drawImage(img, 0, 0, img.width, img.height);
    const descriptor = await faceapi.computeFaceDescriptor(c);
    if (descriptor) descriptors.push(descriptor);
  }
  const t1 = performance.now();
  console.log('time:', t1 - t0, 'average:', (t1 - t0) / dir.length, 'descriptors:', descriptors.length);
}

main();

time: 1687.4630840420723 average: 76.70286745645784 descriptors: 22

[vladmandic]

a) what's the backend used? i'm only assuming it's the correct one tjfs-node b) try newer port of faceapi which uses tfjs 3.9.0 and tensorflow 2.6: @vladmandic/face-api
(i've been maintaining it for the past year since original is no longer maintained)

[danies8]

a)yes "dependencies": { "@tensorflow/tfjs-core": "^0.13.11", "@tensorflow/tfjs-node": "^0.1.17",

[danies8]

Hi, I download it throgh npm (@vladmandic/face-api) const faceapi1 = require('../../node_modules/@vladmandic/face-api/dist/face-api.node.js'); and modified the sample ? and got this error: (node:24184) UnhandledPromiseRejectionWarning: TypeError: tf8.io.weightsLoaderFactory is not a function at FaceLandmark68Net.loadFromDisk (D:\Camera\Backup Face Recognition\1\face-recognition\node_modules\@vladmandic\face-api\src\NeuralNetwork.ts:106:31) at Ob What do i miss?

[vladmandic]

remove @tensorflow/tfjs-core as it's already included in tfjs-node and install latest @tensorflow/tfjs-node - you have obsolete version 0.1.17, latest is 3.9.0

[danies8]

I removed the core and installed the lates version of tenserflow and got this error: What do i miss ?

• Building TensorFlow Node.js bindings node-pre-gyp install failed with error: Error: Command failed: node-pre-gyp install --fallback-to-build 'node-pre-gyp' is not recognized as an internal or external command, operable program or batch file.

[vladmandic]

that means your nodejs on widows is not the best - it cannot run binary bindings as part of package installation process
imo, i'd reinstall node and npm and make sure that node-gyp compiler is installed npm i -g node-gyp

and make sure it's at least node v14, but i'd recommend v16

[danies8]

I installed node-pre-gyp + installed lates version of tenserflow.

[vladmandic]

did you load tf before faceapi? new version requires tf to be loaded first as it cannot bind to binary distribution and it would be bad to do a fake bind.

add const tf = require('@tensorflow/tfjs-node'); at the start, before you load faceapi

[danies8]

Now I uninstall both libraries and install tf before faceapi.

[danies8]

still same error

[vladmandic]

I said load, not just install? Can you show the code where you're loading both tf and faceapi?

[danies8]

I got this error : on thus line:const descriptor = await faceapi1.computeFaceDescriptor(c);

PS D:\Camera\Backup Face Recognition\1\face-recognition> npm run start Debugger attached.

face-recognition@1.0.0 start ts-node-dev ./src/app.ts

Debugger attached. [INFO] 19:00:19 ts-node-dev ver. 1.1.8 (using ts-node ver. 9.1.1, typescript ver. 4.4.2) Debugger attached. Platform node has already been set. Overwriting the platform with [object Object]. cpu backend was already registered. Reusing existing backend factory. 2021-09-14 19:00:22.463783: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 14/09/2021 19:00:22:503 app.js - info : Server start running at http://localhost:6060 14/09/2021 19:00:22:975 app.js - info : Server start listening at port:6060 14/09/2021 19:00:22:977 app.js - info : Server is start loading bbt face matcher tensorflow (node:7716) UnhandledPromiseRejectionWarning: TypeError: forwardFunc is not a function at D:\Camera\Backup Face Recognition\1\face-recognition\node_modules\face-api.js\node_modules\@tensorflow\tfjs-core\src\engine.ts:586:31
at D:\Camera\Backup Face Recognition\1\face-recognition\node_modules\face-api.js\node_modules\@tensorflow\tfjs-core\src\engine.ts:424:20
at Engine.scopedRun (D:\Camera\Backup Face Recognition\1\face-recognition\node_modules\face-api.js\node_modules\@tensorflow\tfjs-core\src\engine.ts:435:19)

[vladmandic]

you just posted the same app code - where is the load part?
e.g., where are your require or import statements?
judging by the log, you're loading tfjs twice and causing conflict

[vladmandic]

why would you assign function to a global variable? and i have no idea how your project is structured and what is a controller you mention - but anyhow, that is not a face-api question
my advise? refactor your code not to use global variables ever

[danies8]

1.I want to load all images when server init , and later consume it in other routes ? How do you refactor it? I used for compare activity. I tried to used node-cache instead, I create singelton from it to share but without success. With global is working. TypeError: Method get TypedArray.prototype.length called on incompatible receiver [object Object] 2.What does _distance mean ?

{
     "bestMatchPerson": {
        "_label": "unknown",
        "_distance": 0.7806719517998486
    }
}

3. Do i need both "@vladmandic/face-api": "^1.5.2", + "face-api.js": "^0.22.2", in pacage json ?
4. You said:(i've been maintaining it for the past year since original is no longer maintained) What do you mean face-api.js is no longer maintined ?

[vladmandic]

1. i get the goal, but dumping stuff into global is always a bad idea. anyhow, your choice
2. distance is euclidean distance. take a look at https://github.com/vladmandic/human/wiki/Embedding#face-similarity on how it's calculated.
3. no, only one, never both. loading both causes conflicts you've already seen.
4. correct. see notes at https://github.com/vladmandic/face-api#note

[vladmandic]

why are you importing with strange relative paths?

faceapi = require('../../node_modules/@vladmandic/face-api/dist/face-api.node.js');

can be just

faceapi = require('@vladmandic/face-api');

or if you want to be specific

faceapi = require('@vladmandic/face-api/dist/face-api.node.js');

never import installed modules with relative paths inside node_modules

and enclose your js code in markdown code quotes so it's actually readable - i cannot read this '```js ...

> Your package is maintained ?

Yes. Just check git comit history to for any package to see when is the last update and how often it receives updates.

> When i removed "face-api.js": "^0.22.2", from pacage json i got this error, i do not used in code at all.
> [ERROR] 11:01:21 Error: Cannot find module '@tensorflow/tfjs-core'

Something in the code is still referencing `@tensorflow/tfjs-core`, but it's no longer installed  
I'd clean `node_modules` and re-run `npm install` and then check both your code and `package-json.lock` file to see where is it referenced  

My `face-api` for nodejs references just `@tensorflow/tfjs-node` package

> Regading Eucliean distance, when i can say us good match and when is not ?

Match is never 100%, so think of Euclidean distance as `chance of match`  
What is a threshold for a good match? That is a personal choice - try what values are good for you as it highly depends on input images  

[danies8]

2. Can you please give example of: a. When node server is loaded than load all images(10,000 is reasonable) using createBbtFaceMatcher and insert into best data structure(not global variable as you said. b. When i register a new image is update the images data structure for opertaion in c. c. When i compare it used the images data structure for best match

[danies8]

3. I fixed to use:

const faceapi = require('@vladmandic/face-api')

[danies8]

5.I markdowned code quotes in all comments as you asked-:) 6.I used these weights wjen working with face-api.js, i need to change it to other files ?

[vladmandic]

If i got "_distance": 0.7 (above) what does it mean and if I got 0.3(below) what does it mean ?

distance 0 means identical and 1 means completely different. default threshold for matching is distance < 0.6

btw, just choose one model, don't test for both
ssd is typically better than tiny in everything, tiny is included only for very low power devices

Can you please give example of: a. When node server is loaded than load all images(10,000) (createBbtFaceMatcher ) insert into best data structure(not global variable). b. When i register a new image is update the images data structure for opertaion in c. c. When i compare it used the images data structure for best match

this is a general architecture question, perhaps this discussion is close to what you're looking for: https://github.com/vladmandic/human/discussions/138 and https://github.com/vladmandic/human/discussions/145

also see notes in https://github.com/vladmandic/face-api/issues/51 for general json save and notes on on multi-process analysis to fully utilize available hardware https://github.com/vladmandic/face-api/issues/20

10,000 images is not a huge number, but it's getting there and it's worth it to architect right - you don't want to keep 10k objects in global variable ever. and what if it grows? solution should be such that you can switch to db store if needed so there is no need to ever load all in-memory

[danies8]

1. How do I insert the 10k object to sql db?(createBbtFaceMatcher ), I defined coulum as json type and later insert record for all labeledFaceDescriptors or for each labeledFaceDescriptor ?
2. When I register i need to add new record to db and on compare i need to load all t from db 10k records is it fast ? and later do?

   let  descriptorFromDb = go to db and bring all descriptor;
   let faceMatcher = new faceapi.FaceMatcher(descriptorFromDb );
   const singleResult = await faceapi
       .detectSingleFace(referenceImage)
       .withFaceLandmarks()
       .withFaceDescriptor()
   const bestMatchPerson =  global.faceMatcher .findBestMatch(singleResult.descriptor)

[vladmandic]

How do I insert the 10k object to db?(createBbtFaceMatcher ), how i defined in db ?

That is far beyond this conversation. For a proper solution, I suggest using mongodb module
For a prototype, storing object as on disk is a start (fs.writeFileSync('db.json', JSON.stringify(myObject))) Anything is better than a) calculating them again-and-again on each startup, b) keeping in global variable

When I register i need to add new record to db and on compare i need to load all t from db 10k records is it fast ?

findBestMatcher is simply a for loop through that runs matches for all records and returns one with lowest distance - not more and not less

So if you read records from DB yourself and pass them to match method them, get the same thing without needing to have all records in memory all the time. Of course, you don't want to read neither all records nor one-by-one, it should have some sane disk paging

But again, this is far beyond this conversation.

[danies8]

@vladmandic Thank you for you answers. But I don't understand your final solution if you please elaborate ?

[vladmandic]

architecture is a personal choice. plus solution architecture is far beyond this scope
im sharing here a short writeup, but please don't go further with architecture questions - lets limit the scope to library issues

first, i suggest using an actual database. you might start with nedb-promises which allows you to easily switch to mongodb in the future if database grows a lot
structure is at least two different object tables: images and faces (image can have more than one face, don't assume it's always just one)

on server startup

1. initialize database and pass handle to other module (such as module that handle uploads) so they can share access
2. enumerate image files and for each check if its in the db or if its modification time is different than one in db
3. if necessary process image and store file descriptors to faces object table and map from which file it came from
4. store file details in files object table

then on how to handle uploads

1. on image file upload, trigger steps 3 and 4 from above
2. run face match on all records to get best match if above threshold
3. based on the best match find image file and return it to client

optionally

1. run file system watcher so it automatically finds new or modified files so you don't need to restart server to process additional images
2. use worker process pool for maximum performance so you can process as many images in parallel as you have cpu cores

[vladmandic]

Do you mean to save photos in DB not in file system ? and when you say file is photo or image ?

No, save photos in filesystem. Only data about the photo goes into database.

How long it will take to get best match for 10k images in your solutions? seconds minites....

Same. Issue is memory usage and scalability of the solution.

These red tiny files are unneccerary:

All of them are - you don't need to keep a local copy at all as they are already part of the module and you can load them from module folder. You only need to host them somewhere when you're dealing with browser solutions that don't have access to node_modules.

[danies8]

OK-:) 1.Why you use nedb-promises/mongodb becase sql db cant save objects(descriptors)?

2. How long it will take to get best match for 10k images in your solutions? you said same=> like loading process that take 5 minutes not few seconds ?

[vladmandic]

serously - the message tells you exactly what is wrong - check your paths. default is ./model, but you need to override that with what matches your project.

anyhow, i'm out of this thread.

[danies8]

Thanks for answer. I'm new with this lib, sorry about questions :) I understood that i do not need my weights lib it come from your package. I tried both model && ./model and is not working .

[vladmandic]

Need full path here - something like 'node_modules/@vladmandic/face-api/models'

[danies8]

Thanks, In my case await faceDetectionNet.loadFromDisk(path.join(__dirname, '../../node_modules/@vladmandic/face-api/model'));

[danies8]

Can you please answer: 1.Why you use nedb-promises/mongodb becase sql db cant save objects(descriptors)? The reason of the questions we used sql server and if it possible not to use another type of db.Untill is must.

2. How long it will take to get best match for 10k images in your solutions? you said same=> like loading process that take 5 minutes not few seconds ?

[vladmandic]

SQL database can store objects, but they are non transparent blobs. You want to go with SQL, go for it, nothing against it, just it will be more work and more overhead for no gain. Mongo stores objects, not records. It's object database. So JS object that you get during processing can be directly stored and accessed without any transforms.

Regarding performance, when I say "same" I mean that we'll designed DB layer will not add performance degradation. What is ideal performance to calculate Euclidean distance for 10k records? Few seconds.

[danies8]

Thanks alot again:)

[vladmandic]

yes
but it's not like you need to define a strict schema in mongodb - it will just as well take any object you pass to it

[danies8]

You mean like this?

const mongoose = require('mongoose');
let Schema = mongoose.Schema;
let FacesSchema = new Schema({
    face: {
        type: []
        required: true
    },
 });
module.exports = mongoose.model('faces', FacesSchema)

[danies8]

Hi, Why you said:" In faces table (image can have more than one face, don't assume it's always just one)" Please add code samples.

[vladmandic]

isn't it obvious? photo or video can have no persons in it or any number of persons

now, if you're calling

await faceapi.computeFaceDescriptor(img);

like in your example, you're calculating descriptor for a photo, not for a face - and you always get single descriptor per photo as you never processed photo into individual faces (if input photo is nicely cropped photo of a single face, it's ok, but what if its not?)

that method is intended for already processed (cropped) images of faces

but if you do it like this

const result = await faceapi
  .detectAllFaces(img, optionsSSDMobileNet)
  .withFaceLandmarks()
  .withFaceDescriptors()

then face-api will detect faces and process landmarks and descriptors for each detected face. and result is array.

[vladmandic]

Isn't that for you to test what fits your use case best?!

[danies8]

The reason that I asked this question is that I saw in face-api.js that all pictures are cropped photes and gray? for example: and I asked how your face api will want the pictures(format/size/color) to make best match, i need the match for black people ? Can you please answer ? I test the code above on black people, but is it the right conclusion: to crop the image and to make it gray, does it harm the the bestMatch function ?

[vladmandic]

details?

• input is converted to tensor and resized (and padded to keep aspect-ratio) to fixed 512x512 rgb with 8-bits per channel
• detected faces are each cropped and resized to square 150px for ssd or 112px fortiny`
• face is normalized to grayscale before further processing

so if you want to save

• inputs: save them to 512px long dimension and keep them rgb
• faces: just save face as you get it from face-api. your way of converting to grayscale is not conversion as all, its stripping channels and ruining the image

DO NOT crop faces and convert to grayscale manually - let face-api do that
you don't know exactly how to crop the image since that is determined by detector process

[danies8]

Thanks for your answer, I will fixed it. 1.What about image type jpeg, png ? 2.The client has bitmap from camera and converted to base 64 string and the node server server get in route base 64 string, where i save them to 512px long dimension ?

[vladmandic]

how should jpg vs png matter?!? image is decoded into bitmap for any processing, like with any image processing software and regarding base64 strings, they are from helper classes so they can be directly attached to image elements in browser i prefer to never deal with base64 when in nodejs. i use image library like canvas and save bitmap to jpg. see examples in my repository.

sorry, it's really time to put this thread to rest - i tried to help, but this is sooooo off-topic. i will not reply here anymore.

[danies8]

@ vladmandic Thank you very much, you help me -:) Regarding base64 strings, the client and server are not in same project, there are separated, the client react send the bitmap as base64 string to node js api, i don't know other way to send image through api except as base64 string.

[danies8]

isn't it obvious? photo or video can have no persons in it or any number of persons

now, if you're calling

await faceapi.computeFaceDescriptor(img);

like in your example, you're calculating descriptor for a photo, not for a face - and you always get single descriptor per photo as you never processed photo into individual faces (if input photo is nicely cropped photo of a single face, it's ok, but what if its not?)

that method is intended for already processed (cropped) images of faces

but if you do it like this

const result = await faceapi
  .detectAllFaces(img, optionsSSDMobileNet)
  .withFaceLandmarks()
  .withFaceDescriptors()

then face-api will detect faces and process landmarks and descriptors for each detected face. and result is array.

I used your advise and it works-:) Except some cases that match not work 100% precent.