Automattic / kue

Kue is a priority job queue backed by redis, built for node.js.
http://automattic.github.io/kue
MIT License
9.45k stars 862 forks source link

Pausing while active jobs not updating job state #840

Open EmileSpecs opened 8 years ago

EmileSpecs commented 8 years ago

Hi,

So pause and resume works in that when I have a queue of jobs and pause them, no other jobs in the queue are processed until I resume, perfect.

The problem is that the jobs that are "active" and successfully complete never get updated to "complete" and stay "active" even after resume.

My job queue processes 10 jobs asynchronously, while they're processing (I'm just using a timeOut for testing) I pause them and resume them and this is then my output.

Job #895 got queued of type: communication-sms
Job #896 got queued of type: communication-sms
Job #897 got queued of type: communication-sms
Job #898 got queued of type: communication-sms
Job #899 got queued of type: communication-sms
Job #900 got queued of type: communication-sms
Worker is paused...
Worker is paused...
Worker is paused...
Worker is paused...
Worker is paused...
Worker is paused...
done! 895
done! 896
done! 897
done! 898
done! 899
done! 900
Worker has resumed...
Worker has resumed...
Worker has resumed...
Worker has resumed...
Worker has resumed...
Worker has resumed...

After the "done!" output a "Job #x completed!" output is expected in the console but these jobs are not updated to have completed even though they do complete (as seen by "done!").

So I end up having lots of "active" jobs in my queue even though they have completed! This is surely not intended behaviour? And if it is, then it shouldn't be...

Any help on this issue? Thanks!

bymodude commented 8 years ago

I believe I am encountering the same issue, I simply want to process jobs one at a time but wait for 5 seconds after each job of that type before processing the next one, so I have this example code:

Jobs.process('testJob', function(job, ctx, done) {
    setTimeout(function() {
        done();
        ctx.pause(500, function(err) {
          setTimeout(function() { ctx.resume(); }, 5000);
        });
    }, 1000);
});

what happens is that when ctx.pause() is called it never completes the job in redis, that means I never receive the Jobs.on('complete') event even though the job obviously completed

if I dont call ctx.pause() directly after the done(), e.g I delay it by 1000ms, then I do get the event, so it seems as soon as ctx.pause() is called it will no longer update active jobs as complete when they are marked as done

for my use case I would already be happy with a hint on how to properly implement the pattern "process jobs one at a time but wait x seconds in between", but I do agree with OP that this behaviour of .pause() is not expected

behrad commented 8 years ago

Yes, it seems as a race between pause and done