Job Dependency Supported?

[rontendo-jp]

You guys did an amazing job on this. Thanks for making it open sourced.

One question, I know that you are using APScheduler and it doesn't have the capability to say "Job B depends on Job A.... it will not run until Job A is successful". How do you guys address this in your production env? You must have a job that will start after another job finishes.....

Thanks

[wenbinf]

You can structure the code of "Job B" and "Job A" into python functions that are agnostic to ndscheduler. "Job dependency" is achieved by simply calling python functions :)

For our production system, we use ndscheduler together with taskworker, and we also opensource taskworker

When ndscheduler schedules a job, it actually publishes a message to Amazon SQS, then one of taskworker processes would pick up the message from SQS and does the actually work.

When "A runs every 5 minutes and B runs immediately after A", then A is a job scheduled by ndscheduler, while B is not. Both A and B are tasks that run on taskworker. We trigger B from the code of A. Here's the sequence:

1. ndscheduler triggers A and publishes a message to SQS, containing the python class of A and the corresponding arguments passed to the python class in that message.
2. taskworker fetches the message of A from SQS and runs the corresponding python class of A.
3. In the python class of A, it publishes another message to SQS, containing the python class of B.
4. taskworker fetches the message of B from SQS and runs the corresponding python class o B.