Automattic / kue

Kue is a priority job queue backed by redis, built for node.js.
http://automattic.github.io/kue
MIT License
9.46k stars 867 forks source link

Create new queue job in process block of another job, but on different prefixes #1059

Closed fsfarah closed 7 years ago

fsfarah commented 7 years ago

I have a parent job creator and processor. When processing parent job, I need to create a child job. But this job has to be on a different Redis instance and a different prefix altogether. However when I try to add the new child job, it is being added on the parent's queue instead of the child's queue!

Here's the code:

Child Process

'use strict';

const kue = require('kue');
const queue = kue.createQueue({
  prefix: 'child-test'
});

queue.process('from-child', (job, done)=> {
  console.log('job from-child recieved');
  setTimeout(()=> {
    done(new Error('timed out'));
  }, 300);
});

Child Create

'use strict';

const kue = require('kue');
const queue = kue.createQueue({
  prefix: 'child-test'
});

module.exports = {
  add: function(cb) {
    const job = queue.create('from-child', {
      name: 'child job'
    }).save();

    job.on('complete', (result)=> {
      return cb(null, result);
    });
    job.on('failed', (err)=> {
      return cb(err);
    });
  }
};

Parent Process

Notice here how I create a new child job inside the process block

'use strict';

const kue = require('kue');
const queue = kue.createQueue({
  prefix: 'parent-test'
});

const child = require('./child-create');

queue.process('from-parent', (job, done)=> {
  console.log('parent job recieved', job.data);
  child.add((err, result)=> {
    if (err) {
      return done(err);
    }else {
      return done(null, result);
    }
  });
});

Parent Create

'use strict';

const kue = require('kue');
const queue = kue.createQueue({
  prefix: 'parent-test'
});

module.exports = {
  add: function(cb) {
    const job = queue.create('from-parent', {
      name: 'parent job'
    }).save();

    job.on('complete', (result)=> {
      return cb(null, result);
    });
    job.on('failed', (err)=> {
      return cb(err);
    });
  }
};

So in order to initiate I run the following

'use strict';

const parent = require('./parent-create');

parent.add((err, result)=> {
  if (err) {
    console.error(err);
  }else {
    console.log(result);
  }
});

The problem is that I see the jobs from child and parent on the same queue with prefix parent-test instead of being separated into parent-test and child-test resulting in any child job to be stuck.

However, if I rename all the queue prefixes to test it works like a charm.

How can I fix this issue? Thanks

behrad commented 7 years ago

If I guessed it correct, your are depending on two separate Kue instances with different prefixes in the same process, however createQueue currently returns a singleton. So you should run different prefixes in separate node.js processes.