Anyone able to get workerpool to work on Vite?

[stereokai]

I've been pulling my hair on this for a few hours...

How do I set up workerpool in a Vite app to use a dedicated worker?

I tried almost every setting combination I could figure out.

// main.js
import DataGeneratorWorker from "./DataGeneratorWorker.js?worker";

const pool = workerpool.pool(new DataGeneratorWorker());

pool
  .exec("fibonacci", [10])
  .then(function (result) {
    console.log("Result: " + result); // outputs 55
  })
  .catch(function (err) {
    console.error(err);
  })
  .then(function () {
    pool.terminate(); // terminate all workers when done
  });

// DataGeneratorWorker.js 
import workerpool from "workerpool";
// OR
 importScripts("workerpool")

// a deliberately inefficient implementation of the fibonacci sequence
function fibonacci(n) {
  if (n < 2) return n;
  return fibonacci(n - 2) + fibonacci(n - 1);
}
// create a worker and register public functions
workerpool.worker({
  fibonacci: fibonacci,
});

I end up either with two separate instances of workerpool, and the error Error: Unknown method "fibonacci", or unable to call the regular workerpool instance from the worker with importScripts()

[josdejong]

It may be that you'll first have to create a bundle of the worker, and then load that bundle. This is a known difficulty discussed in #189

[stereokai]

Hey, thanks a lot for getting back.

I have made some progress since I asked this question, and after a bit of learning I figured out that's what I need to do.

I'd like to comment though, that it would be nice to be able to use ES module imports in workers, but at the moment it seems workerpool in the browser only supports the "classic" type of work, i.e. with importScripts().

Thanks

[josdejong]

Good to hear. Good point about the ES module. I've opened an issue for that: #342. Will close this issue now.

[stereokai]

That's cool, tanks!

[brunoAmado]

hi sterekai, can you say more about the "classic" way please

[stereokai]

@brunoAmado Yes, it basically means your worker TS/JS file should be a traditional JS script, and not an ES6 module (can't have any import or export statements).

Instead of importing them using ES6 modules, I used https://github.com/syJSdev/rollup-plugin-copy-merge in my Vite config to merge all the separate sources that make my worker, incl. the workerpool script (from node_modules/workerpool/dist/workerpool.js) into a single javascript file.

I then import the compiled worker file in the app like this:

import workerpool from "workerpool"
const workerPool = workerpool.pool(
  new URL("./CompiledWorker.js", import.meta.url).href,
  {
    maxWorkers: 3,
  }
);

You can see the approach I took here: https://github.com/stereokai/multi-charts-comparison You will be interested in the Vite config, the import and finally the worker file using workerpool.

I know that it's ~a little bit~ convoluted, but it's working, and it's the only way I could get it to work both in development and production, because of this issue: https://github.com/vitejs/vite/issues/7569

[brunoAmado]

Thanks you, you are an inspiration and saviour. Will try that even if I'm using Next.js it will be helpfull

[stereokai]

Happy I could help 😊

[Ctrlmonster]

Here is another way that worked for me. Kind of ugly and I had to disable "isolatedModules" in my tsconfig. But yeah, really wanted to use workerpool. From the worker.ts file:

// import your latest workerpool version from a cdn here
importScripts("https://cdn.jsdelivr.net/npm/workerpool@6.3.1/dist/workerpool.min.js");

function fibonacci(n: number): number {
  if (n < 2) return n;
  return fibonacci(n - 2) + fibonacci(n - 1);
}

// create a worker and register public functions
// @ts-ignore: ts won't know workerpool here
workerpool.worker({
  fibonacci: fibonacci,
});

Edit: This works as well if you have it installed locally, not sure what's the better option here lol

importScripts("../node_modules/workerpool/dist/workerpool");

[stereokai]

@Ctrlmonster is it working for you on Vite 2? Or Vite 3?

[Ctrlmonster]

@Ctrlmonster is it working for you on Vite 2? Or Vite 3?

Vite 3

[stereokai]

@Ctrlmonster So I thought. This way is the standard Ecmascript way, but it's not supported in Vite 2. At least not in 2.9.1, don't know about later 2.9.x versions (this is what the original issue was about).

[sedghi]

@stereokai I see your repo multichart is not public anymore, is it possible for you to put here the pieces of code that you referenced above? thanks!

[stereokai]

@sedghi Sure. I had to take it down because the project has graduated into a proprietary product. Here is the code I originally linked to earlier:

vite.config.js:

import replace from "@rollup/plugin-replace";
import jscc from "rollup-plugin-jscc";
import copy from "rollup-plugin-copy-merge";

const replaceRules = {};

const jsccRules = {
  values: {
    _DEVELOPMENT: process.env.NODE_ENV === "development",
  },
  options: {
    asloader: false,
  },
};

const copyTargets = [
  {
    src: [
      resolve("workerTask1.js"),
      resolve("workerTask2.js"),
      resolve("workerApi.js"),
    ],
    file: resolve("CompiledWorker.js"),
  },
];

const workerpool = "node_modules/workerpool/dist/workerpool.js";
if (process.env.NODE_ENV === "production") {
  copyTargets[0].src.unshift(
    resolve(...workerpool.split("/"))
  );
}
if (process.env.NODE_ENV === "development") {
  replaceRules["//importScripts()"] = `importScripts("/${workerpool}")`;
}

export default defineConfig({
  plugins: [
    copy({
      hook: "buildStart",
      targets: copyTargets,
    }),
    replace(replaceRules),
    jscc(jsccRules),
//...

app.js:

import workerpool from "workerpool";
const workerPool = workerpool.pool(
  new URL("./CompiledWorker.js", import.meta.url).href,
  {
    maxWorkers: 3,
  }
);

workerApi.js:

//#if _DEVELOPMENT
//importScripts();
//#endif

workerpool.worker({
  task1: workerTask1,
  task2: workerTask2,
});

[sedghi]

@stereokai Really appreciate it. Quick question, did any of your workerTasks have any external dependencies too or not?

[stereokai]

Nothing from node_modules, only "imports" from within the codebase. Whichever kind of dependency you might have, you'll need to use the copy plugin to compose it into the compiled worker, exactly like in my Vite config - where you can see examples for local modules (as in the tasks) as well as an external dependency/npm package (as in workerpool).

[sedghi]

Thanks for the explanation! I'm wondering whether Rollup can do tree shaking based on what piece of the node_module dependency is needed inside the task, and does not bring the whole dependency

[stereokai]

I'm not sure, because this is not really ES/CJS imports per se, it's simple text file composition. But, you could add an earlier build step that imports the node_modules you need using Rollup, compiles it with tree shaking into a separate, temporary bundle, and simply add the bundle path to the source array of the copy plugin. It will copy that bundle into the compiled worker, together with your tasks. You just have to make sure Rollup exposes the imported packages in the global scope and doesn't encapsulate them as a module to import, otherwise your tasks couldn't access their dependencies. I'm sure there's a Rollup option for that.

[VizualAbstract]

For anyone curious, Workerpool has a Vite example up and going.

https://github.com/josdejong/workerpool/tree/master/examples/vite

import WorkerURL from './worker/worker?url&worker'
const pool = workerpool.pool(WorkerURL, {
    maxWorkers: 3,
    workerOpts: {
        // By default, Vite uses a module worker in dev mode, which can cause your application to fail. Therefore, we need to use a module worker in dev mode and a classic worker in prod mode.
        type: import.meta.env.PROD ? undefined : "module"
    }
});

[wszgrcy]

Can I add URL type input? Using URL. href will result in errors in cjs TypeError: The worker script or module filename must be an absolute path or a relative path starting with './' or '../'. Wrap file:// URLs with new URL. Received "file:///home/xxx/my-project/ts-test/worker-test/b.mjs"

import { join } from 'path';
import { Worker } from 'worker_threads';
import { pool } from 'workerpool';
(() => {
  let a = join(__dirname, 'b.mjs');
  //   console.log(a);
  // new Worker(new URL(a, 'file:')); //success
  let instance = pool(new URL(a, 'file:').href);
  instance.exec(''); //error
})();

now I run with

import { join } from 'path';
import { pool } from 'workerpool';
(() => {
  let a = join(__dirname, 'b.mjs');
  let url = new URL(a, 'file:');
  let instance = pool(url.href, {
    onCreateWorker(arg) {
      return { ...arg, script: url as any };
    },
  });
  instance.exec('');
})();

[josdejong]

Not sure, but maybe you can use a dynamic import like pool(await import('./b.mjs')) ?