Tensors resulting from face detection seemingly cannot be accessed and disposed

[zmk-punchbowl]

Issue Description

We're using the following code to do face detection on image files. Note, the getHuman function either initializes human, if needed, or returns the already initialized object. Also the logs have been added for debugging.

  const buffer = fs.readFileSync(imgFile);

  const tensor = (await getHuman()).tf.node.decodeImage(buffer);
  const res = await (await getHuman()).detect(tensor);

  // Dispose all tensors
  if (res?.face) {
    console.log("res.face", res.face);
    res.face.forEach(async (f: FaceResult) => {
      console.log(`Disposing face tensor`, f.tensor);
      (await getHuman()).tf.dispose(f.tensor);
    });
  }

  if (tensor) {
    console.log(`Disposing tensor`, tensor);
    (await getHuman()).tf.dispose(tensor);
  }

  console.log("numTensors", (await getHuman()).tf.engine().memory().numTensors);

  return res.face as FaceResult[];

We are doing face detection, which means that in our config, we have:

  face: {
    enabled: true,
    detector: {
      enabled: true,
      rotation: false,
      maxDetected: 10,
      minConfidence: 0.6,
    },
    mesh: { enabled: false },
    iris: { enabled: false },
    description: { enabled: true },
    emotion: { enabled: false },
  },

My understanding is that this will result in tensors attached to the FaceResult. And indeed we can see numTensors increasing, the more we call this function. However, res.face.tensor appears to be undefined in all cases. The only tensor we're able to see exists and dispose of, is the initial one obtained from decodeImage.

So what we see is numTensors increasing, along with the memory allocated to the process, which, given enough workload, will run out of memory eventually.

Steps to Reproduce

Run the above code on a series of images, over and over.

Expected Behavior

The number of tensors remains steady after they are used and disposed of. Memory usage also remains stable.

Environment

Node.js on an M1 Mac

• Human library version? 3.2.1
• Built-in demo or custom code? Custom
• Browser or NodeJS and version (e.g. NodeJS 14.15 or Chrome 89)? Node 20
• OS and Hardware platform (e.g. Windows 10, Ubuntu Linux on x64, Android 10)? M1 Mac, also see same behavior on Amazon Linux

[zmk-punchbowl]

After reading #232 a bit more, I noticed that we are not setting face.detector.return to true. When I do this, then the face.tensor is no longer undefined and we can dispose it. However, the tensor count and memory still increase.

[zmk-punchbowl]

Also note that we were on 3.2.1. Upgrading to 3.2.2 didn't seem to change things.

[zmk-punchbowl]

Some more information. I noticed we had this in our config.

deallocate: false,

I assume this is okay, since we are attempting to manually deallocate using .dispose? Anyway, I tried setting this to true. Memory usage seemed maybe slightly affected? However I still ended up with ~1,000 tensors more than when I started (exact same as when deallocate is false). That's not expected after using dispose on everything, is it?

Lastly, the only other places we use human is calling match.similarity. I assume there aren't any tensors from this that we need to clean up?

[zmk-punchbowl]

After more testing, I have to correct the above observation. Changing to deallocate: true does help. Memory increases at first (perhaps as expected), but over a longer period of time, it stays stable, rather than increasing unbounded.

This solves the immediate problem for us, which is good. However, I still think there may be an issue where manually deallocating the tensors does not provide the same (or similar) results as having deallocate: true.

[vladmandic]

this is pretty well documented from your side, not much i can add right now - I'll take a look and try to reproduce when I'm back from my travels in 2 weeks.

[vladmandic]

i wrote a full reproduction used your base code and i see no leaks, with or without return=true

const fs = require('fs');
const H = require('../dist/human.node.js');

const config = {
  debug: true,
  modelBasePath: 'https://vladmandic.github.io/human-models/models/',
  body: { enabled: false },
  hand: { enabled: false },
  face: {
    enabled: true,
    detector: {
      return: true,
      enabled: true,
      rotation: false,
      maxDetected: 10,
      minConfidence: 0.6,
    },
    mesh: { enabled: false },
    iris: { enabled: false },
    description: { enabled: true },
    emotion: { enabled: false },
  },
};

const human = new H.Human(config);
let i = 0;

async function detect(imgFile) {
  const imgBuffer = fs.readFileSync(imgFile);
  const imgTensor = human.tf.node.decodeImage(imgBuffer);
  const res = await human.detect(imgTensor);
  if (res?.face) {
    console.log(`loop=${i} faces=${res.face.length} tensors=${human.tf.engine().memory().numTensors}`);
    res.face.forEach((f) => human.tf.dispose(f.tensor)); // only needed if return:true
  }
  human.tf.dispose(imgTensor);
  return res.face;
}

async function loop(imgFile) {
  for (i = 0; i < 99; i++) {
    await detect(imgFile);
  }
}

loop('../samples/in/group-2.jpg');

loop=0 faces=3 tensors=253
...
loop=98 faces=3 tensors=253

if the issue persists, write a full self-contained reproduction (e.g. reference to getHuman as simple as it may be mean that my reproduction is different by definition), update here and i'll reopen.