Kue is not adding jobs to redis instance

[lfcgomes]

Hello guys,

I've been struggling with an issue that I dont have any idea how to solve or why it happens.

When I use Kue with a local instance of redis, everything is just fine, and work as it should. The problem is when I moved everything to my staging environment, Kue went crazy. Sometimes jobs are not added to the queue, and sometimes they are added but they're not handled by workers (just get stuck in inactive state).

The first jobs (one or two) that I add to the queue are done, but after a few minutes they're not added anymore or are not handled.

Does anyone have any idea or have faced a similar problem? It's very strange that everything is working locally, but not with a remote instance of redis.

Thanks!

[behrad]

There may be some point in code where a default createClient is being called on node_redis! This is a simple guess, Can you listen on error event and give me more details on what may be happening...?

[lfcgomes]

I'm sorry, but I don't think that I understand your answer... Where do you want me to listen on error event?

By the way, I'm already listening the error event when calling .save(), but error is always undefined

queue.create('JOB', { folderID: '/' }).save(function(err){ console.log(err);});

[behrad]

1) please do:

queue.on( 'error', ... )

2) and stop your redis instance on localhost, so that if Kue is creating any localhost connections, you face some errors to see if thats the case

[lfcgomes]

I did what you asked for.

var q = kue.createQueue({
    prefix: 'q',
    redis: {
        port: config.get(env+':queue:port'),
        host: config.get(env+':queue:host'),
        db: config.get(env+':queue:db'),
        auth: config.get(env+':queue:password')
    }
});

q.on( 'error', function( err ) {
    console.log( 'Oops... ', err );
});

1. there is no error being printed in the console.
2. I also stopped the local redis to see if anything would pop up.

After a while, without adding any job to the queue, jobs are not being added anymore, until I restart my nodejs app :\

Any other idea?

[behrad]

Strange! I can't figure out what is happening until you show me a code that can reproduce your situation then.

[lfcgomes]

I have a function A that schedules background job (send forgot password email). This is the code I use to do that:

var queue =             require(configDir + '/libs/kue');

queue.create('EMAIL', {
    emailType: 'forget',
    firstName: req.user.firstName,
    lastName: req.user.lastName, 
    resetPasswordToken: req.user.resetPasswordToken,
    destEmail: req.user.username
}).removeOnComplete(true).save();

Then, I have a kue.js where I create the redis connection and import the file that deals with the email stuff.

/*jshint node: true */
'use strict';

var kue =               require('kue');
var configDir =         process.cwd() + '/config/';
var config =            require(configDir + 'config');
var env =               config.get('NODE_ENV') || 'development';

var q = kue.createQueue({
prefix: 'q',
redis: {
    port: config.get(env+':queue:port'),
    host: config.get(env+':queue:host'),
    db: config.get(env+':queue:db'),
    auth: config.get(env+':queue:password')
    }
});

q.on( 'error', function( err ) {
console.log( 'Oops... ', err );
});

require('/job/email)(q);

module.exports = q;

and this is my email.js:

module.exports = function(jobs) {
    jobs.process('EMAIL', 20, function(job, done){

        var data = job.data;
        switch(data.emailType){
            case 'forget':
                emailTypeForget(data, done);
                break;
            default:
                log.error('emailtype: '+data.emailType+' not found');
                done(new Error('emailtype '+data.emailType+' not found'));
        } 
    }); 
};

Also, in my last experiments, after listening on error event, this appeared in my consoled: `Oops... [Error: Redis connection to XXX.XXX.XX.XXX:6379 failed - connect ETIMEDOUT]``

[behrad]

and that shows Kue is not being able to connect to your redis instance

[lfcgomes]

But now the same problem occured without having that error :\

[lfcgomes]

It looks like after a few moments of inactivity, Kue "loses" the connection to my redis instance. But, at the same time, no error pops up when saving the job to it, which make me believe that it is still connected.

[behrad]

I have upgrade node_redis lib to latest. Can you check if there's any improvements on redis connection handling in Kue 0.10?

[moredure]

Have same problem. After restarting redis server kue not adds jobs to the queue and not process them. v0.11.5

[behrad]

Can you test v1 branch @mikefaraponov ?

[buffpojken]

So, I've just found this and have the same issue.

Do I understand this correctly - if a single createClient is used on node_redis before kue.createQueue is created, kue will ignore the redis-config passed to it at creation time?

[jgervin]

Having the same issue. I define a remote Redis on AWS and it keeps using Redis on localhost?

[wmikega]

bummer no solution. We are having some issues with Kue and both local and AWS Redis. Not much in my logs other than these messages which makes me think Kue is in a frozen state:

node: 'JOB STATUS ---- The Job with id: 8 has been queued ---- file name: undefined', Thinking about dropping Kue and getting SQS setup.