search

vsivsi / meteor-job-collection

A persistent and reactive job queue for Meteor, supporting distributed workers that can run anywhere.
https://atmospherejs.com/vsivsi/job-collection
Other
388 stars 68 forks source link

Outside of event-loop #240

Open janat08 opened 7 years ago

janat08 commented 7 years ago

So if I were to run very expensive computation, what would happen is that it would suspend as soon as something should be done in event-loop?

vsivsi commented 7 years ago

Is this is a general node.js question? Not specific to job-collection?

In general, because node.js is single threaded, expensive calculations should be avoided specifically because the block the event-loop. There is no automatic "suspending" of computation in node.js.

Where job-collection can help is by helping to move expensive computations out of your server's event loop and onto separate vanilla node.js worker processes (either on the same physical server, or on other machines reachable from it). This is the point of the meteor-job npm package. You must manage these worker processes yourself, and must establish and maintain their DDP connections to the server(s) hosting the job-collection. But after that, job-collection will distribute the work, and collect progress, status, results, etc.

Hope that helps.

janat08 commented 7 years ago

fine. I think ill just try scaling horizontally first and try and break up this computation. I'm hoping that there's a way to look at server load before accepting work? Mongo server uses significant amount of ram for each connected meteor.js (DDP, I suppose), looking to double hosting servers as workers is a valid form of laziness.