Schedule and processing multiple jobs on Heroku

[promisenxu]

I'm trying to use kue-scheduler to recurringly run some functions. Detailed situations:

• I have multiple functions to be invoked, at different intervals/time
• Each function to be processed by Kue is a http request
• I don't really care about the status or results of each job recorded by Kue.

What I have done is scheduling all the jobs in the main process and process all the jobs in a separate work process.

However I discovered that only the last job called with "queue.process" will be invoked. This seems to be a hard limit for kue, i.e. only one job can be processed in each work instance. Is this true? If so, any recommendation on how I can implement the code to fit my needs?

[lykmapipo]

@promisenxu

• I have multiple functions to be invoked, at different intervals/time In this scenario you mean you want to delay execution and perform it later or you want to schedule some works to be perform in regular interval of time?.

  • If it just delay the work use job.delay().
  • If you want to schedule to be performed at regular intervale use Queue.every(, job);

• Each function to be processed by Kue is a http request If I get you here is that you want to queue a work to be performed in background or other process. The way forward it to have an instance of kue in your main process that you can use to dispatch work to background process or other process

  • In your main process

    function(request, response) {
     //pre process your request
      ...
     let job = queue.create(<type>, <data>);
     job.save();
    
    //respond
    }

  • I don't really care about the status or results of each job recorded by Kue I can advice to update progress status or store the status of the work using the worker process so that you can implement job resume logic in case of failure.

• kue concurrency is limited with how many process you attach in processing the work plus the number of worker you allow per process. So far I found it perform its intended job very well. Cross check of there is any other process where you have invoked queue.process.

Hope it helps.

[promisenxu]

@lykmapipo Thanks for the detailed response.

From what you said:

kue concurrency is limited with how many process you attach in processing the work plus the number of worker you allow per process. So far I found it perform its intended job very well. Cross check of there is any other process where you have invoked queue.process.

So it is true that ONLY one job can be processed per worker instance on heroku? I'm trying to call queue.process() multiple time with different jobs on the same worker instance, not calling queue.process() on multiple worker instances for the same job.

[promisenxu]

@lykmapipo

To further clarify, what I'm trying to do is:

In the main process:

var queue = kue.createQueue({redis:process.env.REDISCLOUD_URL, skipConfig:true});
var job1 = queue.createJob('job1', {});
queue.every('5 seconds', job1);
var job2 = queue.createJob('job2', {});
queue.every('10 seconds', job2);

In the worker process:

var queue = kue.createQueue({redis:process.env.REDISCLOUD_URL, skipConfig:true});
queue.process('job1', function(job, done) {
    // Do something
    console.log('job1 finished);
    done();
});
queue.process('job2', function(job, done) {
    // Do something
    console.log('job2 finished);
    done();
});

With the above example, only job2 will ever be processed. I want job1 and job2 (and many other jobs, in my production environment) to both be processed. I'm not sure how to achieve that.

[promisenxu]

@lykmapipo I seemed to have solved my problem - I was using a for loop in my actual code to schedule and process all the jobs. There were a scope problem with the variables. I fixed that and everything is working fine now. Thank you for your help!