Thread-based workers

[markfinger]

This is an interesting read on threading in Node http://neilk.net/blog/2013/04/30/why-you-should-use-nodejs-for-CPU-bound-tasks/

In general, it seems more practical these days:

• https://github.com/audreyt/node-webworker-threads
• https://github.com/xk/node-threads-a-gogo

Though, there is still the usual caveat with JS workers that they only evaluate scripts, so you need to build libraries with webpack/browserify.

The big benefit of threads (vs node's cluster or fork) would be reduced memory usage. A common problem that we've hit on servers (both during CI and deployment) is that we blow out the VM's memory limit and the build process dies halfway. Threads might be a solution to this problem.

[markfinger]

86

[markfinger]

Didn't take much poking through the issues for each of the threading libraries before issues start emerging (lack of event loop and primitives - setTimeout, etc).

That being said, I did come across one exciting little thing: https://github.com/nodejs/node/pull/2133#issuecomment-222754118

[markfinger]

Something I've been struggling to handle cleanly is how to handle multiple phases with workers.

The use-case that's rolling around involves an initial transformation/dep-analysis phase, followed by contextual rewrites of the file (dep paths, inlining assets, etc).

Fundamentally, it looks like a question of how to handle resumable/iterative jobs across separate processes?

Probably need some notion of jobs where we can handle multiple iterations of work on assets. A key part of this would be preserving references to expensive data (ASTs, for example) and trying to minimise transport overhead. Also need to cover invalidating records.