ml5 errors when loading two different models simultaneously

[ziyuan-linn]

Hi all,

Thank you @Tinazz333 for discovering and letting me know about the issue below.

When loading handpose and facemesh (and probably other models as well) simultaneously, a runtime error will occur:

RuntimeError: Aborted(Module.arguments has been replaced with plain arguments_ (the initial value can be provided on Module, but after startup the value is only looked for on a local variable of that name))

The following code in p5 will trigger this error:

let handpose;
let facemesh;
preload() {
  handpose = ml5.handpose();
  facemesh = ml5.facemesh();
}

A trick that seems to fix the problem is forcing the models to load one by one like this:

let handpose;
let facemesh;
preload() {
  handpose = ml5.handpose(() => {
    facemesh = ml5.facemesh();
  });
}

However, this still causes the browser to throw a bunch of warnings:

The kernel '...' for backend 'webgpu' is already registered

[ziyuan-linn]

Upon further investigation, it appears that the issue is caused by mediapipe runtime. I experimented using tfjs runtime using the code below and the error went away.

preload() {
  handpose = ml5.handpose({ runtime: "tfjs" });
  facemesh = ml5.facemesh({ runtime: "tfjs" });
}

I came across this thread on the mediapipe repo discussing a similar issue. The suggested solution involves loading the models one by one.

"currently the initializers don't play nicely together. But you shouldn't need anything complicated-- just adding await before the .initialize() calls (specifically await selfieSegmentation.initialize()) should be sufficient."

One potential solution for us is to implement a queue within ml5 for model initialization calls using the mediapipe runtime. This queue would handle the loading of models one by one in a synchronous manner.

I'd appreciate any thoughts or further insights.

[alanvww]

I recently worked on a project and encountered the same problem. I didn't notice that this can be solved by switching to another runtime, but here is my solution:

async function modelLoader() {
  // Create a function that returns a promise for loading faceMesh
  const loadFaceMesh = () =>
    new Promise((resolve) => {
      faceMesh = ml5.faceMesh(options, () => {
        console.log("faceMesh loaded");
        resolve();
      });
    });

  // Create a function that returns a promise for loading handPose
  const loadHandPose = () =>
    new Promise((resolve) => {
      handPose = ml5.handPose({ maxHands: 20 }, () => {
        console.log("handPose loaded");
        resolve();
      });
    });

  // Await the promises in sequence to ensure proper order of execution
  await loadFaceMesh();
  await loadHandPose();
}

function preload() {
  modelLoader();
}

I would definitely like to see if the library itself can have some built-in logic for solving this problem.

[eyaler]

small typo in:

let handpose;
let facemesh;
preload() {
  handpose = ml5.handpose(() => {
    facemesh = ml5.facemesh();
  });
}

should be:

preload() {
  handpose = ml5.handPose(() => {
    facemesh = ml5.faceMesh();
  });
}