search

humanmade / Cavalcade

A better wp-cron. Horizontally scalable, works perfectly with multisite.
https://engineering.hmn.md/projects/cavalcade/
Other
515 stars 46 forks source link

Jobs can hang if they output too much data #37

Closed joehoyle closed 7 years ago

joehoyle commented 7 years ago

After what I can only describe as an intense session of debugging with @rmccue we discovered that some jobs were hanging when the pipe's buffer filled up from stderr / stdout of a running job. This could be something like a cron job doing an excessive amount of error_logging.

The buffer size is relatively small by default, and I believe the runner only reads from the pipe once the child process has completed. This means the sys call to write to the pipe will hang when full.

As a "workaround" we removed the error_log calls from the job, and if need be it's possible to cat /proc/{process id of wp-cli}/fd/2 to flush the pipe buffer.

rmccue commented 7 years ago

Currently, we exhaust the streams on shutdown. This should instead be part of the worker pipe check in the loop.