humanmade / Cavalcade

A better wp-cron. Horizontally scalable, works perfectly with multisite.
https://engineering.hmn.md/projects/cavalcade/
Other
517 stars 46 forks source link

When recurring jobs fail, future iterations fail + WordPress still thinks it'll run #23

Closed dd32 closed 4 years ago

dd32 commented 7 years ago

Ran into an interesting scenario today where a daily recurring job failed last week and it wasn't queued again / re-run.

In the cavalcade_jobs table, the status was 'failed'

This ended up in a situation where WordPress thought the job was queued and Cavalcade didn't attempt to run it.

In this case, I manually reset the status field for the jobs in question to waiting and it continued on it's way.

dd32 commented 7 years ago

recurring jobs set status = 'waiting' after successful runs, so it runs again.

I'm unsure if the failure is the runner or the plugin (or both). It would make sense that the plugin would only show jobs listed as status = 'waiting', which will cause wp_next_scheduled() not to show a date for failed jobs. It would also make sense for the Runner not to set the job as failed for a recurring job.

Changing the plugin to only return waiting jobs would mean that if a recurring job failed, there'd be multiple entries in the cavalcade_jobs table for that blog/hook - I'm unsure if that's a problem though.

ocean90 commented 7 years ago

In this case, I manually reset the status field for the jobs in question to waiting and it continued on it's way.

When doing that then for some reasons the job is running every second for a while. Is that because it's re-running all the jobs which didn't run before?

bildschirmfoto 2016-11-13 um 15 08 15
dd32 commented 7 years ago

When doing that then for some reasons the job is running every second for a while. Is that because it's re-running all the jobs which didn't run before?

Yes, unless you update the nextrun field too. for most jobs it doesn't matter, but that job which was running every minute had a fair bit of iterations to catch up on (as the next run is calculated by nextrun = nextrun + schedule)

rmccue commented 7 years ago

Correct; if you change it back to waiting, Cavalcade will continue running it normally, and it doesn't skip cycles. If you have a task that runs every minute, and it stops for five minutes, it will have to run 5 times before it catches back up.

dd32 commented 4 years ago

I think the intention of #27 was to fix this, which I'm assuming it did as it hasn't been an issue on WordPress.org in years.