Automattic / kue

Kue is a priority job queue backed by redis, built for node.js.
http://automattic.github.io/kue
MIT License
9.46k stars 867 forks source link

Only move job into Active when condition is met - Kue node js #1073

Open smdevpod opened 7 years ago

smdevpod commented 7 years ago

Is the following possible?

I was wondering if its possible to hold the job in a queued state before putting it into active only when it meets a condition. Example:

This is what the jobs look like =

var job = queue.create('myQueue', {
    from: 'process1',
    type: 'testMessage',
    data: {
        msg: 'Hello world!'
    }
}).save(function(err) {
        console.log('Job ' + job.id + ' saved to the queue.');
});

This is my queue.process =

queue.process('myQueue', 4, function(job, done) {
  console.log('job', job.data)
  handleMessage(job, done); 
});

As soon as the job is picked up my queue.process, it is moved to active state. However, I only want to move the job into active state, when there is something in the database, like so:

 myDatabase.find(function(err, docs) {
   if(docs){
      //move the job to active state
   } else {
      // hold the job in a queued state and query the dbs again in 10 
      // seconds
   }
})

Is this possible? Any help would be appreciated as Im still trying to figure out if KUE would be appropriate for me

behrad commented 7 years ago

No, Why aren't you creating the job when your condition is met? let it be queued inside kue when your condition is met, and it will be processed ASAP

adityak368 commented 7 years ago

+1

jbreadner commented 7 years ago

+1

I'm using Kue to manage telnet / FTP actions against physical devices; I'd like only one job to be active at any given time for each physical device, but for jobs to be active on multiple devices concurrently.

With the answer given by behrad, we'd have to implement some sort of buffer in front of Kue to store all pending jobs, then only release a job from the buffer to Kue if there is not an active job in Kue for that device. It would be much more powerful to add some capability to Kue to help it decide whether it should start processing an inactive job.

m4rcelofs commented 7 years ago

@jbreadner same use case here (#1098), did it by having one queue per device (with concurrency one). The consumer adds one processor to each queue by calling queue.types.