josdejong / workerpool

Offload tasks to a pool of workers on node.js and in the browser
Apache License 2.0
2.06k stars 147 forks source link

Question: complex objects to worker #399

Closed vanodevium closed 1 year ago

vanodevium commented 1 year ago

Hello! First of all, thank you very much for this incredible library!


I have a question that has probably been asked 100 times, but I haven't found an answer that works for my case.

I have a pool of 10+ workers. Each of them is a function that uses an instance of puppeteer. And I need to somehow make it so that I can transfer exactly one instance of the puppeteer, because at the moment (due to the fact that the pool creates workers very quickly), I cannot achieve a singleton.

My worker file looks like:

const workerPool = require("workerpool");
const Worker = require("./worker");

const worker = new Worker(/* maybe I can make singleton exactly at this place? Or pass instance? No? */);

async function processPage(options) {
  return worker.process(options);
}

workerPool.worker({
  processPage: processPage,
});

Please help. I will be grateful for any answer! Thanks!

josdejong commented 1 year ago

Some ideas:

vanodevium commented 1 year ago

So far, the only way I've come up with is as follows:

I am creating an instance of puppeteer and waiting it by await and only after that I initialize the pool of workers in which I only connect to that instance.

I could not come up with more options. Thank you for your reply. Until we close, maybe someone else will recommend some method to me.

Thank you!

josdejong commented 1 year ago

👍

I'll close this issue now, feel free to comment with more options or reopen if needed.