worker_threads compatibility?

[guybedford]

Understood if this is off-topic, and feel free to close, but I was just wondering if you'd considered compatibility with Node.js worker_threads here at all?

It seems like the sort of thing that would be generally useful for Node.js optimization as well, in having some kind of --node output, since you're already in the game of worker builds.

Although perhaps, a translation layer from web-style workers to Node.js-style worker_threads would be the better route as its own project.

Otherwise do you know anyone working on this problem / who considers this important?

[developit]

This seems worthwhile! It would also be interesting to see if it could be handled automatically by output.target=node.

I've been following Node's workers stuff and would be interested in collaborating if you're looking at a shim layer.

[guybedford]

My primary concern here right now is Node.js going stable with worker_threads before these workflows as discussed above have been explored at all, by anyone.

If there are issues / frictions I think it would be important to get that into the Node.js feedback process before worker_threads are unflagged as experimental and the API is cemented.

Perhaps all will be fine though, in which case, great.

This project seems the closest to the edge of future worker build workflows that I can find right now - thanks for being open to have the discussion here.

[developit]

Sorry for falling off this thread - I agree. I am going to try to find some time to set up tests against worker_threads so I can see what the delta is between what we output today and what needs to be supported.

I'm curious if you envision worker-plugin as being the right place to "patch" worker_threads to include things like addEventListener?

[guybedford]

I'm curious if you envision worker-plugin as being the right place to "patch" worker_threads to include things like addEventListener?

Definitely, would be good to DM further about this if you have a moment.

[andywer]

Node 12 is out, shipping worker_threads without the feature flag. Might be the right time to get back to this? 😉

[fridgerator]

Also interested in this feature!

[developit]

@guybedford and I ended up coming up with a longer-term plan for this, which should be easy to support in Worker Plugin.

In the interim, I believe the following should work today with Worker Plugin:

const { Worker } = require("worker_threads");

const worker = new Worker("./my-worker.js", { type: "module" });

[guybedford]

@developit that seems like a good approach. It's worth noting though that there are a lot of differences between worker threads and native workers. See eg https://github.com/nodejs/node/issues/30682#issuecomment-561010832.

The simple difference from your example is "./my-worker.js" doesn't work as a worker reference in Node.js as it is a process.cwd-relative path not a module-relative URL.

The list is basically:

1. Use new URL('./worker.js', import.meta.url) as the input to the Worker call (as soon as https://github.com/nodejs/node/pull/31664 lands)
2. worker.addEventListener is worker.on in Node.js
3. self.addEventListener in a worker is require('worker_threads').parentPort.on in Node.js
4. In Node.js worker event data e.data must be accessed without the .data part.

I think this is simply too much compat friction for your approach mentioned above to work for any user that doesn't have half a day to mess around with this stuff....

[developit]

Good point, agreed! I think the shim module is probably a better way forward. Need to release it..

[mihanizm56]

any updates?

[DarrenCook]

We're using worker-plugin (and I now need it to work from node too). I was hoping to find a comparison with https://github.com/developit/web-worker

But having read this thread I'm wondering if web-worker is basically the solution being described here? Would love to hear from someone if there are downsides to switching from worker-plugin to web-worker.

[developit]

@DarrenCook I believe the solution is to use both. worker-plugin generates the code for the worker, and web-worker provides a Worker global that can be used by that code.

You likely need to do something like this:

import WebWorker from 'web-worker';

globalThis.Worker = WebWorker;

new Worker(
  new URL('./worker.js', import.meta.url),
  { type: 'module' }
);