vsivsi / meteor-job-collection

A persistent and reactive job queue for Meteor, supporting distributed workers that can run anywhere.
https://atmospherejs.com/vsivsi/job-collection
Other
387 stars 68 forks source link

How to deal with Workers, Collections and Fibers? #170

Closed bluepuma77 closed 8 years ago

bluepuma77 commented 8 years ago

I really hope this is my last question for meteor-job-collection. I would like to title it "From callback hell to fiber hell".

Job is working, I can save and receive jobs. Within the Worker I would like to insert an item into a Collection - and of course get an error Error: Meteor code must always run within a Fiber. Try wrapping callbacks that you pass to non-Meteor libraries with Meteor.bindEnvironment.

I am not familiar with Fibers and Meteor.bindEnvironment and from my point of view I don't even need the environment, I don't want to do any user checks or the like, the Worker is fully trusted.

So is there a quick fix for var queue = JobsQueue.processJobs( to get everything running within a required Fiber? Or do I need to encapsulate every callback with the Worker with an own Fiber?

Is this challenge the reason the documentation has the client outside of the Meteor framework?

vsivsi commented 8 years ago

Fibers were Meteor's worst idea, IMO. Great in theory for simple projects, but falling somewhere on the scale between persistent irritation and nightmarish hellscape for more complex ones.

When running in the Meteor server environment, the worker function in processJobs() is called from within a Fiber. If you call non-Meteor node.js async functions, you must wrap the callback in Meteor.bindEnvironment(). There's really no way around this basic fact of life in the Meteor server environment.

In the photo gallery sample app you can see that I had to do it twice for the exec calls into graphicsmagick for creating thumbnails: https://github.com/vsivsi/meteor-file-job-sample-app/blob/master/sample.coffee#L536-L586

I have a little helper function that makes bindEnvironment a little less annoying, but beyond that, it is what it is...

betterBindEnv = function (func) {
   if ((func !=== null) && (func !=== undefined)) {
      return Meteor.bindEnvironment(func, function (err) { throw err; });
   } else {
      return func;
   }
};
bluepuma77 commented 8 years ago

Perfect, thank you for the detailed response!