Scheduled job gets 'stuck' when Urkund service down

danmarsden / moodle-plagiarism_urkund

Ouriginal plagiarism plugin for Moodle (previously called Urkund)

https://ouriginal.com/

12 stars 19 forks source link

Scheduled job gets 'stuck' when Urkund service down #87

Closed thepurpleblob closed 8 years ago

thepurpleblob commented 8 years ago

I know how bad this report is but you may have a bright idea so here goes...

We have seen this twice. The Urkund service goes down for a short time (we know because Urkund tells us) and a day or so later the debug screen is full of loads of Pending jobs. When we look at the scheduled tasks we see that the Urkund send process has not run since round about the time the service was unavailable. The receive task seems fine.

We kill the cron processes (which appear to have been stuck there) and it all starts to run again.

We have no idea what the cron was doing when it was 'stuck'.

danmarsden commented 8 years ago

interesting - I've heard anecdotally of this happening in other cases. I wonder what the timeout process is for the moodle core cron tasks? - I'll do some further investigation (next week probably) and see what controls the timeout.

danmarsden commented 8 years ago

I'm wondering if this is due to CURLOPT_TIMEOUT - I think it's set to 0 by default in PHP so it never times out during a curl call? - I'll add a few timeout values to the curl calls to see if that helps.

thepurpleblob commented 8 years ago

My installation has this as the default...

CURLOPT_TIMEOUT => 13

It's not the only timeout though...

CURLOPT_CONNECTTIMEOUT => 78

My first thought is that 78 seconds is quite a long time to wait. Anyway, I'll get the server guys to run the same bit of code to check the default constants on one of the production front-ends just in case they vary for some reason.

EDIT: Production box constants are the same.

danmarsden commented 8 years ago

how did you get those default values? - curl_getinfo() just returns the values used in the last call (and many functions in Moodle seem to set CURLOPT_TIMEOUT)

danmarsden commented 8 years ago

ah - I see, those values are actually the "constants" for the options - for some reason curl uses integer values to name each variable but has some constants to help: eg "curl_setopt($fp, CURLOPT_TIMEOUT, "value")

danmarsden commented 8 years ago

I think I've addressed this with the last patch - I've updated the plugins db with the new version as well. Let me know if something similar occurs again and we can look into it again! - thanks!

thepurpleblob commented 8 years ago

We're going to do a Moodle upgrade next month so I'll probably leave it until then.

Meanwhile, we'll run the new version on our test/dev boxes and see how we get on.

Thanks, as ever :-D

thepurpleblob commented 8 years ago

This happened again (we're still running the pre-fix version of the plugin) but this time it was with the \plagiarism_urkund\task\get_scores task instead....

danmarsden commented 8 years ago

cool - I added timeout values to all the curl calls (including the one that does the score check) so hopefully we won't see this with the upgraded plugin.

although - I've seen a similar task locking issue on another unrelated plugin - I suspect we might need some way to generally manage/view locked tasks in core at some point.