Also splits up runner class into a few component classes.
The distributed lock implementation is weak, but according to the redis docs it’s good up to the point that you need a multi-master cluster, which is a loooong ways up the demand chain. Once this is working maybe it can be subbed out for a full redlock.
This is needed to address #80 and other comments/suggestions about parallelization more completely. The new structure to Runner allows it to spawn out a dozen or so executors and a single coordinator -- though that behavior itself will be handled in a different PR. Each executor will be able to pop jobs quickly but only the coordinator will be allowed to prep them for execution, and enqueue from delayed or scheduled plans.
Coordinators need a distributed lock to ensure that when multiple processes are running there is no redundancy about scheduling CRON tasks, etc. Currently this must be disabled with a config option on all but one worker, which causes instability when that single worker dies and cannot be recovered.
[x] make the functionality opt-in; default=off until it's release-tested
Also splits up runner class into a few component classes.
The distributed lock implementation is weak, but according to the redis docs it’s good up to the point that you need a multi-master cluster, which is a loooong ways up the demand chain. Once this is working maybe it can be subbed out for a full redlock.
This is needed to address #80 and other comments/suggestions about parallelization more completely. The new structure to Runner allows it to spawn out a dozen or so executors and a single coordinator -- though that behavior itself will be handled in a different PR. Each executor will be able to pop jobs quickly but only the coordinator will be allowed to prep them for execution, and enqueue from delayed or scheduled plans.
Coordinators need a distributed lock to ensure that when multiple processes are running there is no redundancy about scheduling CRON tasks, etc. Currently this must be disabled with a config option on all but one worker, which causes instability when that single worker dies and cannot be recovered.