search

josdejong / workerpool

Offload tasks to a pool of workers on node.js and in the browser
Apache License 2.0
2.04k stars 148 forks source link

Make sure stats are always consistent when handling a task result #411

Open rbartoli opened 9 months ago

rbartoli commented 9 months ago

Hi, thank you for this great library.

I'm using it with the following syntax:

const pool = workerpool.pool(__dirname + '/import-worker-fast-v3.js', {
  minWorkers: 'max',
  maxWorkers: 8,
  workerType: 'thread',
});

During execution of a loop, I check pool.stats() and I get the following output:

active: 7 
pendingTasks: 9786 
busyWorkers: 7 
idleWorkers: 1 
totalWorkers: 8

One worker is always idle. Am I doing something wrong?

josdejong commented 8 months ago

Thanks for reporting. How can we reproduce your issue?

rbartoli commented 8 months ago

This example reproduces the issue:

package.json

{
  "name": "workerpool-bug",
  "version": "1.0.0",
  "main": "index.js",
  "type": "module",
  "dependencies": {
    "workerpool": "^8.0.0"
  }
}

index.js

import workerpool from "workerpool";

const pool = workerpool.pool("./worker.js", {
  minWorkers: "max",
  maxWorkers: 8,
  workerType: "thread",
});

function checkCompleted(result) {
  const poolStats = pool.stats();

  console.log(
    `pendingTasks: ${poolStats.pendingTasks} active: ${poolStats.activeTasks} busyWorkers: ${poolStats.busyWorkers} idleWorkers: ${poolStats.idleWorkers} totalWorkers: ${poolStats.totalWorkers}`
  );
}

for (let i = 0; i < 100; i++) {
  pool
    .exec("processChunk", [])
    .then(checkCompleted)
    .catch(function (err) {
      console.error(err);
    });
}

worker.js

import workerpool from "workerpool";

async function processChunk() {
  return new Promise(function (resolve, reject) {
    setTimeout(function () {
      resolve(Math.random() * 100 + Math.random() * 100);
    }, 1000);
  });
}

workerpool.worker({
  processChunk: processChunk,
});

The output I get is the following:

> node .\index.js
pendingTasks: 91 active: 8 busyWorkers: 8 idleWorkers: 0 totalWorkers: 8
pendingTasks: 90 active: 8 busyWorkers: 8 idleWorkers: 0 totalWorkers: 8
pendingTasks: 89 active: 8 busyWorkers: 8 idleWorkers: 0 totalWorkers: 8
pendingTasks: 88 active: 8 busyWorkers: 8 idleWorkers: 0 totalWorkers: 8
pendingTasks: 87 active: 8 busyWorkers: 8 idleWorkers: 0 totalWorkers: 8
pendingTasks: 86 active: 8 busyWorkers: 8 idleWorkers: 0 totalWorkers: 8
pendingTasks: 85 active: 8 busyWorkers: 8 idleWorkers: 0 totalWorkers: 8
pendingTasks: 84 active: 8 busyWorkers: 8 idleWorkers: 0 totalWorkers: 8
pendingTasks: 84 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 83 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 82 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 81 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 80 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 79 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 78 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 77 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 76 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 75 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 74 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 73 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 72 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 71 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 70 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 69 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 68 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 67 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 66 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 65 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 64 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 63 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 62 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 61 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 60 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 59 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 58 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 57 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 56 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 55 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 54 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 53 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 52 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 51 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 50 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 49 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 48 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 47 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 46 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 45 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 44 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 43 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 42 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 41 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 40 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 39 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 38 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 37 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 36 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 35 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 34 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 33 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 32 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 31 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 30 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 29 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 28 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 27 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 26 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 25 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 24 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 23 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 22 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 21 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 20 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 19 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 18 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 17 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 16 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 15 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 14 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 13 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 12 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 11 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 10 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 9 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 8 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 7 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 6 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 5 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 4 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 3 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 2 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 1 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 0 active: 7 busyWorkers: 7 idleWorkers: 1 totalWorkers: 8
pendingTasks: 0 active: 6 busyWorkers: 6 idleWorkers: 2 totalWorkers: 8
pendingTasks: 0 active: 5 busyWorkers: 5 idleWorkers: 3 totalWorkers: 8
pendingTasks: 0 active: 4 busyWorkers: 4 idleWorkers: 4 totalWorkers: 8
pendingTasks: 0 active: 3 busyWorkers: 3 idleWorkers: 5 totalWorkers: 8
pendingTasks: 0 active: 2 busyWorkers: 2 idleWorkers: 6 totalWorkers: 8
pendingTasks: 0 active: 1 busyWorkers: 1 idleWorkers: 7 totalWorkers: 8
pendingTasks: 0 active: 0 busyWorkers: 0 idleWorkers: 8 totalWorkers: 8

I would expect to have a busyWorkers: 8 and an idleWorkers: 0 for the entire execution (except for the last run).

Correction: from this output I noticed that during the first "run" all workers are working as expected.

josdejong commented 8 months ago

Thanks for sharing this code, I can reproduce your output.

To check whether all workers are busy, you can calculate how long it should take to execute all tasks (100 * 1 sec / 8 workers = 12.5 sec), and measure how long it actually takes, that is indeed 12.5 sec and a little bit extra (time is needed to spin up 8 workers).

What's going on here that you log the statistics right after a task finished, and before a new task is started. Therefore you see 7 workers busy instead of 8. Only the first round things work a bit different: apparently in that case the stats come in when the worker is already marked busy on the next target. This could be related on the order of to two callbacks listening on the same Promise: your callback, and the callback of the workerpool that will start a next task. It would be nice to make this work consistent, if anyone is interested to look into this please let me know.

To "solve" your issue, you can change

.then(checkCompleted)

to:

.then(() => setTimeout(checkCompleted))

Then you'll always see 8 workers busy.

rbartoli commented 8 months ago

Thanks @josdejong, your explanation makes total sense.

Unfortunately at the moment I don't have the time to investigate the consistency issue with the callbacks but feel free to leave the issue open some someone else may look into it.

josdejong commented 8 months ago

👍

I'll adjust the title.