mikz
commented
9 years ago We are using init_worker_by_lua which is executed when new worker is started. So if old crashes and new is started, it should restart the crontab.
What might be happening is that the new worker does not enqueue the job, because the job is already enqueued. The real issue is what we will do with more than 1 worker.
I don't know how feasible this is, but it seems that it's possible that the worker in charge of the crontab sometimes dies and doesn't wake up. We must investigate if this is a real issue, and devise a way to either make it less possible to fail (i.e. move all tasks to other workers, and leave the crontab worker as a "queue manager" only) and fault tolerant (for example, "check and wake" the crontab process when a new request is received, and 5 minutes have passed since the last check and wake).