vsivsi / meteor-job-collection

A persistent and reactive job queue for Meteor, supporting distributed workers that can run anywhere.
https://atmospherejs.com/vsivsi/job-collection
Other
388 stars 68 forks source link

Choosing which job to do #179

Closed openqubit closed 8 years ago

openqubit commented 8 years ago

I am looking at this sample code for doing a sample job

var workers = Job.processJobs('myJobQueue', 'sendEmail',
      function (job, cb) {
        // This will only be called if a
        // 'sendEmail' job is obtained
        var email = job.data; // Only one email per job
        sendEmail(email.address, email.subject, email.message,
          function(err) {
            if (err) {
              job.log("Sending failed with error" + err,
                {level: 'warning'});
              job.fail("" + err);
            } else {
              job.done();
            }
            // Be sure to invoke the callback
            // when work on this job has finished
            cb();
          }
        );
      }
    );

At this point var email = job.data; // Only one email per job where the worker is acquiring a job,can we pass a parameter to tell the worker to only pick jobs where say, approval status is approved approvalstatus: "approved" or status = 1 for instance?.

The status or approvalstatus shall be stored alongside other job data like

var job = new Job(myJobs, 'sendEmail', // type of job
      // Job data that you define, including anything the job
      // needs to complete. May contain links to files, etc...
      {
        address: 'bozo@clowns.com',
        subject: 'Critical rainbow hair shortage',
        approvalstatus: 'approved',
        message: 'LOL; JK, KThxBye.'
      }
    );
vsivsi commented 8 years ago

Hi, no. Job-collection does not inspect the job data for any purpose. The mechanism provided for determining which jobs to run first is to use the job.priority() setting. You can set (and change) the priority based on any criteria you might want to apply. It is completely general.

The alternative (allowing arbitrarily complex and changing rules to determine job priority) would be extremely inefficient because it would be difficult to index the database effectively. It would also be error prone, because corner cases would enter the priority scheme (what happens if there is no approvalStatus, or if it isn't a known value or even a string?)

The solution is to work out a mapping of the relevant data states to an integer priority value (0 is normal, negative values run sooner, positive values run later). Job-collection will respect that priority value when choosing which jobs to assign to workers, and it will do so efficiently because the lookup is fully indexed within the database.

openqubit commented 8 years ago

Thank you for your elaborate answer. That answered the questions i was having.