I overlooked a small but important detail about how periodic jobs are enqueued in #108. The "last run" timestamp was stored in-process, which was fine when the coordinator was manually selected by config parameter. The bug is, when the distributed lock is enabled, all active workers will enqueue each periodic job each time the interval is up.
The fix is to store the periodic job last_run timestamp in a metadata block.
[x] what should the expiry on the metadata block be?
[x] tests for distributed enqueue of periodic jobs
I overlooked a small but important detail about how periodic jobs are enqueued in #108. The "last run" timestamp was stored in-process, which was fine when the coordinator was manually selected by config parameter. The bug is, when the distributed lock is enabled, all active workers will enqueue each periodic job each time the interval is up.
The fix is to store the periodic job last_run timestamp in a metadata block.