search

lykmapipo / kue-scheduler

A job scheduler utility for kue, backed by redis and built for node.js
246 stars 47 forks source link

Optional Disable of Listening for Job Schedule Events #72

Closed gmcnaught closed 7 years ago

gmcnaught commented 7 years ago

I would like to run a multi-process distributed deployment of Kue-Scheduler and I'd like to be able to have a separation of duty between processes which schedule the next jobs, and processes which pick up and handle work.

lykmapipo commented 7 years ago

@gmcnaught So we have to disable job schedule events on the process(main) that schedule them and enable them on the process(worker or background) that process them.

lykmapipo commented 7 years ago

@gmcnaught I think we should start by looking on how kue handle multi-process then we can take motivation from it.

lykmapipo commented 7 years ago

@gmcnaught I made you collaborator. Thanks for your much thoughts and contributions. Feel free to push this project to maturity.

lykmapipo commented 7 years ago

@gmcnaught Because kue-scheduler use key-expiry events to schedule reccur job wont it be a block for us to scale it to multiple process with separation of responsibilities.

gmcnaught commented 7 years ago

Thank you for adding me, I think this is a great project and it is fun to assist on.

Regarding your comments, I believe this should follow the examples of how kue works: https://github.com/Automattic/kue#parallel-processing-with-cluster

Kue, and kue scheduler both use redis exclusively to coordinate what process is building the scheduler so if it is safe to run across multiple processes, we should be able to spread that out across multiple servers on the LAN.

Essentially what I was thinking is this as my use case:

1) You could have some set of processes (or even servers!) that their sole job is to catch the key-expiry event for scheduling and schedule the next job run. Now when

2) Other processes connecting to the same kue instance would only process work on the queue.process - that way they don't catch all of the scheduling events and they are - in theory - more efficient in picking up work evenly.

Perhaps it would make sense to do like the option of whether to configure the key-expiry on kue instantiation:

kue.createQueue({
   skipConfig: true
   scheduleJobs: false //default to true?
});
lykmapipo commented 7 years ago

@gmcnaught Perfectly explained...and it make sense.

So if the scheduleJobs config provided then the process will be listening to expiry events and queue schedules job for next run. Is it?

Thanks.

lykmapipo commented 7 years ago

Implemented. See v0.7.0 release. You will have to pass an option worker to set your current kue-scheduler as worker or not. If woker:true you instance will process the job otherwise not.