queue jobs getting terminated

[futtta]

re. the mailchain; it looks like (at least for 1 user, potentially more) the queue processing job gets terminated (probably after X seconds).

[denydias]

This is not a bug nor a feature. This may be caused by some hosting services and we can't do nothing about it. Our ongoing user story fortunately may end well as his hosting service (WP Engine) provide a way to offload cron elsewhere. Other services may not be that nice and we have no way to force them to.

One thing that may PARTIALLY work is to partition the queue and run it by small batches. The downsides of this approach:

1. It'll introduce and incredibly complex queue control steps into the code, as each job should have an 'i'm running' status being kept and updated on each queue interaction.

2. We will never find out a 'magic number' for how many requests/time the queue should run, as each hosting service have different policies for these kind of things.

So, we're in quite a dead end here.

[denydias]

On queue panel we have some advice for users to deal with this. Our best bet is to improve the text, which actually says:

A WORD ABOUT WORDPRESS CRON: Autoptimize CriticalCSS Power-Up watch the queue by using WordPress Cron (or WP-Cron for short.) It could be faulty on very light or very heavy loads. If your site receives just a few or thousands visits a day, it might be a good idea to turn WP-Cron off and use your system task scheduler to fire it instead.

[futtta]

Let's await feedback from Abhyank :-)

[futtta]

confirmation:

• changing to non-WP cron did not help
• the processes are indeed terminated because they were running for too long (1m)

given PHP by default comes with a 30s max_execution_time and given the fact other hosts will probably have similar limitations we will need to work around this somehow @denydias

2 ideas which might be simpler then your complex queue control:

1. keep the existing queue job, but let it start by doing GETS for pending jobs (that way we will at least have some rules + sooner or later there will be no pending jobs any more and the queue job can continue working with new jobs and doing POSTS)
2. split the queue job up in 2 queue jobs; a. one that handles N jobs and does the POSTS b. another that only handles P jobs and does the GETS

I think (1) is the simplest solution and (2) might be "better". Open for suggestions :-)

[denydias]

Queue sorting is implemented. Order will be: ERROR, JOB_ONGOING, JOB_QUEUED, NEW...

E jobs are not handled by the queue, so there is no harm in making then first and the sort algorithm is quicker that way.

I'm going to implement the request limit to break the queue loop later.

Still waiting for ccss.com reply about keeping GET results for a while as we've discussed by email.

[denydias]

Request limit advanced option ready for testing.

[denydias]

Approved for tagging by @futtta.