Webworkers for io.js

[iankronquist]

What exists and what is lacking

io.js, understandably, does not have any threading primitives, and gives the outward appearance of being single-threaded. There do exist multi-process primitives which act like popen. Threads and processes serve a very similar purpose but are fundamentally different.

Where webworkers fit

Implementing webworkers in io.js would allow developers to share code between the server and the client. Webworkers are designed for exactly the type of expensive and long running work which server side web applications often perform. It should be noted that webworkers are not part of any javascript standards. Instead, they are part of the HTML standard, and in some companies managed by the teams which work on the DOM.

Use in the community

There already exists an NPM package which emulates the behavior of webworkers. On the one hand, this indicates need from the community, on the other hand it indicates that there exists a method of using webworker APIs and sharing code, potentially obviating the need for a native code implementation. However, a native code implementation could be more efficient. In a similar vein, there exists a plethora of packages for managing web workers, using web workers in iframes, and other purposes. Web workers are widely used by the JS community. Members of the TC have indicated interest in integrating web workers into io.js. I cannot find any bugs in the joyent/node bugtracker which reference web workers. This may indicate that web workers are not actually desired by the community, or that webworkers are a newly emerging standard which arose after node.

What belongs in io.js

Obviously the io.js community doesn't want to implement the monstrosities which lurk in the DOM complicated standards which do not serve the focus of the community. Nor do we want to implement things for which there already exist easy to use and powerful alternatives. However, webworkers offer a powerful, standardized API for running simultaneous javascript tasks.

[iankronquist]

Apparently this wasn't a serious proposal. I'll close this.

[bnoordhuis]

For the record, we're more likely to go down the path of full isolate support. That's more or less equivalent to the cluster module but with everything contained in a single process.

It's not on any roadmap because it's mainly an optimization, not a feature. It would allow optimizing operations like sending or receiving binary data between processes; instead of several read/write system calls, just a memory copy or maybe even a pointer swap would do.

We could hack something like that into the current cluster module if libuv supported shared memory; but it doesn't and making it work reliably on all platforms is not trivial.

[phpnode]

@bnoordhuis +1 for that optimisation - the poor performance of IPC in the cluster module is really unfortunate, it means that it's basically impossible to use things like leveldb effectively. The problem: I want to create a data store with node, using leveldb as a storage backend. Leveldb can only be used by a single process, so ideally I'd run that in the master process and share it between N worker processes. The workers would do most of the work, dealing with things like accepting HTTP requests, query parsing, serialisation, validation etc, before sending the query off to leveldb and then formatting the response for the request. Unfortunately the IPC call that requires at the moment totally kills performance. The node leveldb community has come up with things like multilevel to hack around the issue but it's around 10x slower than using leveldb directly. Having a fast way to do this stuff would enable a lot of applications which are currently just too damn expensive.

[basarat]

Is there something fundamental that prevents https://www.npmjs.com/package/webworker-threads from being rolled into iojs/node

[vkurchatkin]

@basarat most threading packages on npm just spin up a new isolate inside a thread with some custom API for message passing. What is desirable is to have full-blown instance instead (event loop, require, etc)

[Samreay]

Has there been any movement on WebWorkers in io.js - any available implementations that work properly?

[Samreay]

Wow, that github thread is massive. I can't figure out if the change got merged in, whether it is still behind an experimental flag or not.

It would be perfect for me - at the moment I have my own pseudo-worker implementation using cluster.fork(), but the performance is roughly ten times slower in message passing that I see when using a web worker inside Chrome.