Automatically removing failed jobs after a certain TTL?

[RodH257]

It seems like this feature doesn't exist at the moment so I thought I'd suggest it, but feel free to point me in the direction of the feature if it does exist.

When a job fails (ie retries are finished and we give up), it sits inside Redis forever until it is manually cleaned up either via the API, or by creating a script that scans through failed jobs and removes them.

It would be great to be able to give failed jobs a TTL, so that after say 1 week they are automatically removed from Redis.

[IvanMMM]

You can do it yourself just in few lines of code

setInterval(function(){
    Kue.Job.rangeByState('failed', 0, 1000, 1, function(err, jobs) {
        Promise.each(jobs,job=>{
            if (job.created_at > Date.now()-7*24*60*60*1000) return;
            job.remove();
        })
    });
},5*60*1000);

[RodH257]

Thanks @IvanMMM - this is our current workaround, our workers will start this interval to scan for failed jobs and remove them.

My thought was that perhaps it could be more efficient if the library could set a TTL on the Redis key when the job fails so it will automatically disappear?

[erickyi2006]

I explored further on the ttl. Apparently the lib has a ttl() function for job. My experiment run the following code job.ttl(10).save(); e.g. job.id = 28 is created. the ttl value showed up in Redis Desktop Manager but the TTL on the message is still -1. if i set the TTL value to 5 in Redis Desktop Manager, it will remove the job after 5 seconds. so the question is that what is purpose of the ttl value in job ? how do we enforce the ttl?

updated if i start the worker to process the jobs, the job that was set by Redis Desktop Manager will still come in as an event. after that, the Kue lib will throw an exception. For now, I will avoid setting the TTL in Redis Desktop Manager.

Error: job "28" doesnt exist
    at Command.callback (.. \node_modules\kue\lib\queue\job.js:178:17)
    at normal_reply (..\node_modules\kue\node_modules\redis\index.js:721:21)
    at RedisClient.return_reply (..\node_modules\kue\node_modules\redis\index.js:819:9)
    at JavascriptRedisParser.returnReply (..\node_modules\kue\node_modules\redis\index.js:
192:18)
    at JavascriptRedisParser.execute (..\node_modules\redis-parser\lib\parser.js:574:12)
    at Socket.<anonymous> (.. \node_modules\kue\node_modules\redis\index.js:274:27)
    at emitOne (events.js:116:13)
    at Socket.emit (events.js:211:7)
    at addChunk (_stream_readable.js:263:12)
    at readableAddChunk (_stream_readable.js:250:11)