search

meteor / meteor-feature-requests

A tracker for Meteor issues that are requests for new functionality, not bugs.
Other
89 stars 3 forks source link

Can compilation be parallelised? #334

Open pmcochrane opened 5 years ago

pmcochrane commented 5 years ago

One of the first thing that amazed me when I first created a node app is that by default everything runs on the one cpu core even if you have dozens more. This single core nature leads to running multiple copies of an app to ensure that you are using up most of the available server resources.

From an outsider looking in, the meteor compilation process seems to be a bunch of sequential build steps - modern to legacy to server to iOS etc. Over the years it has developed very efficient caching to make rebuilds really quite fast. First time builds and bundling however are still pretty slow for most apps.

My question is if it is even remotely possible for the build process to be split into independent processes that can each get their own cpu thread to compile each bundle in the build process?

This may mean compiling the same source file for each build bundle as it may not have been fully compiled in a parallel build. This would reduce the efficiency of caching. However, this multiple compilation behaviour may be offset by being able to compile all bundles concurrently and different threads compiling at different speeds.

I think using multiple processes may give Babel more cpu resources with the minus points being overall memory and disk usage. I think any percentage of speed up for initial build time or bundle time would be greatly appreciated by all meteor devs.

If this is a stupid request then feel free to tell me and I’ll happily close it.

sakulstra commented 5 years ago

was thinking about this as well... not sure how modern/legacy/ios/cordova are related to each other and may access the same cache - but even if they are I guess at least server could be build in parallel?

hexsprite commented 4 years ago

This is now theoretically possible because Meteor now ships with a version of Node that supports worker threads.

nathanschwarz commented 3 years ago

I've developed a multi-core package for meteor : nschwarz:cluster. We are talking about using it for the tree-shaking feature, but we could eventually use it to separate build processes. further more we could already use it to parallelise the multi-arch build with a little work. I will soon add custom master <-> child message broker to enable such features.