vsivsi / meteor-job-collection

A persistent and reactive job queue for Meteor, supporting distributed workers that can run anywhere.
https://atmospherejs.com/vsivsi/job-collection
Other
387 stars 68 forks source link

FIFO queue? #77

Open randomnerd opened 9 years ago

randomnerd commented 9 years ago

Hey there! Is it possible to make a FIFO queue using this package? I need jobs to be executed in the exact same order as they come into the queue. Or do some sort of object locking, so two jobs cant work on same underlying data to prevent collisions. Just an example of use case to explain what we are trying to achieve: We have a collection of objects and onCreate callback which updates other objects in collection on certain conditions. While it finds & updates matching objects, other freshly created objects should wait for that process to be finished if it needs some of objects that previous process is still updating. Not sure if this is clear enough, any suggestions? Thanks!

vsivsi commented 9 years ago

Hi, yes, you should easily be able to do this.

Job-collection allows jobs to be scheduled with a dependency on the successful completion of one or more already scheduled jobs (waiting, running or already finished).

Once a new job object is created, a dependency to a previously scheduled job can be added using:

job.depends([prevJob]);
randomnerd commented 9 years ago

and what happens when a dependant job fails? :)

vsivsi commented 9 years ago

Failed jobs can be configured to automatically retry using job.retry(options). If a failing job exhausts its automatic retries, or if it has a fatal failure, then any dependent jobs will be automatically cancelled. Cancelling a job with dependents also leads to its dependents being cancelled. So the net effect is that the final failure (or cancellation) of a job with dependents will lead to the recursive cancellation of the entire tree of dependent jobs.

Once the failure condition has been remedied, failed jobs can be restarted, and the restart call has a parameter to recursively restart all dependent jobs:

job.restart({ dependents: true })