search

vsivsi / meteor-job-collection

A persistent and reactive job queue for Meteor, supporting distributed workers that can run anywhere.
https://atmospherejs.com/vsivsi/job-collection
Other
388 stars 68 forks source link

Job not repeating properly with Later.js schedule #250

Open joncursi opened 7 years ago

joncursi commented 7 years ago

I don't understand the usage of schedule within repeat. Consider the following code:

job

const job = new Job(jobCollection, 'REFRESH_COIN_COUNTS', {})
  .priority('normal')
  .repeat({
    schedule: jobCollection.later.parse.text('every 1 sec'),
  })
  .save();

worker

const worker = jobCollection.processJobs(
  'REFRESH_COIN_COUNTS',
  {
    pollInterval: 500,
  },
  (job, callback) => {
    console.log(new Date());
    job.done();
    callback();
  },
);

My expectation in writing the above code is that every second, the worker will log to the console since it's polling for new jobs every 500ms, and I'm saying that the job should schedule to run every second.

Instead, this is what I'm getting in console (two logs, every ~15 seconds):

I20170711-05:07:55.807(-4)? Tue Jul 11 2017 05:07:55 GMT-0400 (EDT)
I20170711-05:08:10.833(-4)? Tue Jul 11 2017 05:08:10 GMT-0400 (EDT)
I20170711-05:08:10.859(-4)? Tue Jul 11 2017 05:08:10 GMT-0400 (EDT)
I20170711-05:08:25.901(-4)? Tue Jul 11 2017 05:08:25 GMT-0400 (EDT)
I20170711-05:08:25.917(-4)? Tue Jul 11 2017 05:08:25 GMT-0400 (EDT)
I20170711-05:08:40.999(-4)? Tue Jul 11 2017 05:08:40 GMT-0400 (EDT)
I20170711-05:08:41.016(-4)? Tue Jul 11 2017 05:08:41 GMT-0400 (EDT)
I20170711-05:08:56.094(-4)? Tue Jul 11 2017 05:08:56 GMT-0400 (EDT)
I20170711-05:08:56.110(-4)? Tue Jul 11 2017 05:08:56 GMT-0400 (EDT)
I20170711-05:09:11.171(-4)? Tue Jul 11 2017 05:09:11 GMT-0400 (EDT)
I20170711-05:09:11.185(-4)? Tue Jul 11 2017 05:09:11 GMT-0400 (EDT)
I20170711-05:09:25.741(-4)? Tue Jul 11 2017 05:09:25 GMT-0400 (EDT)
I20170711-05:09:25.754(-4)? Tue Jul 11 2017 05:09:25 GMT-0400 (EDT)
I20170711-05:09:40.837(-4)? Tue Jul 11 2017 05:09:40 GMT-0400 (EDT)
I20170711-05:09:40.849(-4)? Tue Jul 11 2017 05:09:40 GMT-0400 (EDT)
I20170711-05:09:55.930(-4)? Tue Jul 11 2017 05:09:55 GMT-0400 (EDT)
I20170711-05:09:55.946(-4)? Tue Jul 11 2017 05:09:55 GMT-0400 (EDT)
I20170711-05:10:11.006(-4)? Tue Jul 11 2017 05:10:11 GMT-0400 (EDT)
I20170711-05:10:11.020(-4)? Tue Jul 11 2017 05:10:11 GMT-0400 (EDT)
I20170711-05:10:26.092(-4)? Tue Jul 11 2017 05:10:26 GMT-0400 (EDT)
I20170711-05:10:26.108(-4)? Tue Jul 11 2017 05:10:26 GMT-0400 (EDT)
I20170711-05:10:41.183(-4)? Tue Jul 11 2017 05:10:41 GMT-0400 (EDT)
I20170711-05:10:41.196(-4)? Tue Jul 11 2017 05:10:41 GMT-0400 (EDT)
I20170711-05:10:55.757(-4)? Tue Jul 11 2017 05:10:55 GMT-0400 (EDT)
I20170711-05:10:55.773(-4)? Tue Jul 11 2017 05:10:55 GMT-0400 (EDT)
I20170711-05:11:10.846(-4)? Tue Jul 11 2017 05:11:10 GMT-0400 (EDT)
I20170711-05:11:10.859(-4)? Tue Jul 11 2017 05:11:10 GMT-0400 (EDT)

A) I'm so confused on the above.

B) After reading this, it seems like an alternative to schedule on an existing job is to schedule NEW jobs every second (physically insert a new document into the jobs collection in MongoDB). I like that approach, but the only way I can figure out how to do this is wrapping my job in Meteor.setInterval, but that feels sloppy. Surely there's a better way?

joncursi commented 7 years ago

This produces the functionality I'm looking for...

later.setInterval(
  Meteor.bindEnvironment(() => {
    new Job(jobCollection, 'REFRESH_COIN_COUNTS', {}).save();
  }),
  later.parse.text('every 1 seconds'),
);

jobCollection.processJobs('REFRESH_COIN_COUNTS', {
  pollInterval: 500,
}, (job, callback) => {
  console.log(new Date());
  job.done();
  callback();
});

... but I'm not sure if this is the correct way to do it, especially since it seems schedule on repeat is trying to achieve the same thing.

vsivsi commented 7 years ago

If you want very low latency, I recommend using the approach outlined here (instead of polling):

https://github.com/vsivsi/meteor-job-collection#qtrigger---anywhere

That will take care of the "worker" side. On the server side, you should also reduce the time of the "promotion" interval:

https://github.com/vsivsi/meteor-job-collection#jcpromotemilliseconds---server-only

That defaults to 15s and is the reason in the OP that you were only seeing jobs run at that rate.

The final thing account for is that when using repeat, the next repeat of a job will only run after the previous run is done. Repeating jobs will never overlap.