Massive CPU Resources Being Used Auto Cache Engine

[vestaxpdx]

Hi

I have the auto cache engine running on our sites which have up to 5K pages for a couple years.

Despite having a 1 second delay, our server suddenly seems to be grinding to a halt during these runs.

This never used to be the case, is there any known bug about this? Perhaps it is not respecting the delay any more.

Comet Cache Version Pro v170220 WordPress 4.9.4

Stats for just one hour are:

domain1: 8716 hits by comet cache crawler domain2: 7405 hits by comet cache crawler domain3: 2556 hits by comet cache crawler

The above created loads of 50.04 42.52 55.01

We have a dedicated server with a spec of:

2x E5-2620V4 - 16 core (L) RAM : 64 GB RAM DDR4 2133 Mhz Hard Disk : 6 x 240 GB Performance SSD RAID : RAID 10

Running PHP7.0 with PHP OPcache Apache Server MPM: worker

Our server never really struggled before, but the last couple of weeks every day it is melting a bit and causes sites to slow down dramatically.

Many thanks,

[jaswrks]

@vestaxpdx Thanks for the report. Are you running with the HTML Compressor enabled?

I suggest enabling the extended debugging provided by Comet Cache. Note the second option in the dropdown under: Dashboard → Comet Cache → Plugin Options → Enable/Disable

If the HTML Compressor is running, you might take a look at the debug output in the HTML source code for any remote URLs that have stopped responding or are otherwise delayed.

[vestaxpdx]

We do not have the HTML Compressor enabled. But we are using Autoptimize plugin which does the same thing. So it could well be something they have changed but they are really pro on speed I would be surprised if a change they made, made their plugin slower.

Just odd it has suddenly got really bad when it was working fine even back on PHP 5.4 we were running for a bit last year!

Is there anyway the auto cache engine could be running twice in some way? That would definitely create additional load?

Additionally it would be cool if we could easily see a log of when the auto cache is running or something. Not sure if that is already built in or possible? Would help with debugging as this is a difficult one.

I think realistically we might just have to stop using the auto cache engine which will be a shame as our sites seem too large now.

Appreciate any info you have re auto cache logs or notifications we could maybe get.

[jaswrks]

Yes, I suggest installing WP Crontrol so you can review the scheduled WP-Cron jobs and find Comet Cache in the list: https://wordpress.org/plugins/wp-crontrol/

The ACE (Auto-Cache Engine) keeps a log of each URL it processes, in:
wp-content/cache/comet-cache/cache/cc-auto-cache.log

[jaswrks]

Autoptimize plugin which does the same thing ... So it could well be something they have changed

If it downloads data from a remote URL, it could happen even if they didn't change anything. Looking at any network requests that are made when a plugin like that one (or the HTML Compressor) runs would perhaps shed some light.

For example, if one of the JS files your site depends on is actually being downloaded, consolidated, and compressed, and that remote resource has started lagging or is unreachable and timing out, it can result in long-running processes and have the effect of bringing down a server.

[jaswrks]

The ACE guards against this to the extent that it limits the amount of time spent on any single resource. But this may not be the case in all plugins or in all network requests. So it might be worth it to look at the server when the load is high and see how long processes have been running.

[jaswrks]

I think realistically we might just have to stop using the auto cache engine which will be a shame as our sites seem too large now.

Yeah, and that may actually be the right move. At least in the short term, it might make it easier to isolate a problem in logs.

[vestaxpdx]

Many thanks for your detailed comments!