Auto-Cache Engine causing severe slowdown on Multisite Network with large numbers of pages

[scottelkin]

This is what our code tracer says - look how long the code is taking:

quick_cache\plugin::auto_purge_post_terms_cache 4 16,400 ms 40% quick_cache\plugin::auto_purge_home_page_cache 8 5,950 ms 15% quick_cache\plugin::auto_purge_posts_page_cache 8 5,400 ms 13% quick_cache\plugin::auto_purge_post_cache 7 4,530 ms 11%

What should I do about it? Should I downgrade to the last version?

[raamdev]

@scottelkin Thanks for the bug report. It seems that all of the purging routines are taking far longer than they should be. I haven't seen this issue before. A few questions that may help me diagnose the problem while I run further tests:

• Do your posts have a lot of categories or tags?
• Does your site have a large number of categories or tags?
• Would you mind providing a list of active plugins installed on your site? (I'm thinking another plugin might be introducing a conflict somewhere.)
• Does downgrading to the previous version fix the problem for you?

[raamdev]

Also, are you running Quick Cache LITE (from WordPress.org) or Quick Cache Pro?

[scottelkin]

1,2. We have hundreds of sites, so I don't know a way give you an answer if we have a lot of categories or tags.

3. These are the plugins that are turned on network wide. Autoptimize it turned on a site by site basis. And as far as I know, only a few sites use it.
4. I was going to downgrade, but instead I turned off the auto-purging for categores and tags and that made it good enough for now. I also just upgraded to your latest release from this past Friday, so I was hoping that would also make it better.
5. I am using Quick Cache Pro - not LITE.

Add New Users Allows you to bulk create new users on a site and add them to a blog, including the facility to set their role and password on the new site. Version 1.0.7 | By Andrew Billits, Ulrich Sossou

Amazon S3 and CloudFront Automatically copies media uploads to Amazon S3 for storage and delivery. Optionally configure Amazon CloudFront for even faster delivery. Version 0.6.1 | By Brad Touesnard http://bradt.ca/

Amazon Web Services Includes the Amazon Web Services PHP libraries, stores access keys, and allows other plugins to hook into it Version 0.1 | By Brad Touesnard http://bradt.ca/

Autoptimize Optimizes your website, concatenating the CSS and JavaScript code, and compressing it. Version 1.8.5 | By Frank Goossens (futtta) http://blog.futtta.be/

Google Analytics + Enables Google Analytics for your site with statistics inside WordPress admin panel. Single and multi site compatible! Version 3.1.2 | By WPMU DEV http://premium.wpmudev.org/

Multisite Plugin Manager The essential plugin for every multisite install! Manage plugin access permissions across your entire multisite network. Version 3.1.4 | By Aaron Edwards http://uglyrobot.com/

New Blog Templates Allows the site admin to create new blogs based on templates, to speed up the blog creation process Version 2.6.8.2 | By WPMU DEV http://premium.wpmudev.org/

NextScripts: SNAP Pro Upgrade Helper Upgrade/Addon only. NextScripts: Social Networks Auto-Poster Plugin is requred. Please do not remove it. This is not a replacement, just upgrade/addon. Version 1.1.7 | By Next Scripts http://www.nextscripts.com/

NextScripts: Social Networks Auto-Poster This plugin automatically publishes posts from your blog to multiple accounts on Facebook, Twitter, and Google+ profiles and/or pages. Version 3.4.2 | By Next Scripts http://www.nextscripts.com/

Post Indexer Indexes all posts across your network and brings them into one spot – a very powerful tool that you use as a base to display posts in different ways or to manage your network. Version 3.0.5.7 | By WPMU DEV http://premium.wpmudev.org/

Quick Cache Pro WordPress advanced cache plugin; speed without compromise! Version 140725 | By s2Member® / WebSharks, Inc. http://www.s2member.com/

Recent Global Posts Feed RSS2 feed showing global posts - to access feed go to http://yoursite.com/feed/globalpostsfeed Version 3.1.1 | By WPMU DEV http://premium.wpmudev.org/

Simple Sitemaps For Multisite The ultimate search engine plugin - Simply have sitemaps created, submitted and updated for every blog on your site Version 1.1 | By Viper007Bond (Incsub) http://premium.wpmudev.org/

Spam Filter (AVH F.D.A.S Component) Protects your comments by filtering out obvious spam Version 0.1 | By Ve Bailovity (Incsub) http://premium.wpmudev.org/

Stop Spammer Registrations Plugin The Stop Spammer Registrations Plugin checks against Spam Databases to to prevent spammers from registering or making comments. Version 5.9.3 | By Keith P. Graham

Ultimate Branding A complete white-label and branding solution for multisite. Login images, favicons, remove WordPress links and branding, and much more. Version 1.7.2.1 | By WPMU DEV http://premium.wpmudev.org/

Unfiltered MU Adds the unfiltered_html capablitiy to Administrators and Editors so that content posted by users with those roles is not filtered by KSES; Embeds, Iframe, etc. are preserved. Note: If for any reason the unfiltered_html capability is ever lost, simply deactivate, and then reactivate this plugin. Version 1.3.1 | By Automattic http://automattic.com/

WPMU DEV Dashboard Brings the power of WPMU DEV direct to you, it'll revolutionize how you use WordPress, activate now! Version 3.4.5 | By WPMU DEV http://premium.wpmudev.org/

[raamdev]

@scottelkin Just to confirm, you are experiencing this issue within a WordPress Multisite Network, correct?

Also, you said that you're experiencing the slowdown when updating posts: Are you updating posts from the parent site or from one of the child blogs? Or does the slowdown happen from both places?

[scottelkin]

Yes, MU. It was updating posts from the child blog. I don't know that I can update a child post from the master blog.

On Thu, Jul 31, 2014 at 11:04 AM, Raam Dev notifications@github.com wrote:

@scottelkin https://github.com/scottelkin Just to confirm, you are experiencing this issue within a WordPress Multisite Network, correct?

Also, you said that you're experiencing the slowdown when updating posts: Are you updating posts from the parent site or from one of the child blogs? Or does the slowdown happen from both places?

— Reply to this email directly or view it on GitHub https://github.com/websharks/quick-cache/issues/247#issuecomment-50795879 .

/ scott

[raamdev]

OK, thank you for the additional information. That will help me during my attempt to recreate this issue. I will update this issue when I have more information.

[raamdev]

Including the following two screenshots with permission from @scottelkin (after removing any identifying details as requested):

[raamdev]

@scottelkin I have been trying to reproduce the slowness you're seeing, but I have been entirely unsuccessful. I profiled the code and saw no unusual slowness when a post had dozens of tags with cache files to purge upon updating a post.

I also created a multisite test environment with dozens of child blogs, each with dozens of their own posts, and then pre-cached all of the posts in the entire network (2,700+ cache files). When I edited a post in a child blog, which resulted in Quick Cache purging the cache for any associated post terms (categories/tags), there was absolutely no slow-down or difference in purging speed (when compared to a single-site WordPress install).

---

As I've been entirely unsuccessful reproducing the problem you're having, I have to assume that there's another variable at play here, something related to either your hosting environment or one of your other plugins introducing a conflict somewhere that is affecting Quick Cache.

If you can recreate this issue in a clean WordPress install (with no other plugins running) or provide me with any other details that might help me narrow this down, I will be happy to continue looking into this.

[scottelkin]

Ok, thank you for trying. I am running APC. Would that change anything?

On Wed, Aug 6, 2014 at 10:14 PM, Raam Dev notifications@github.com wrote:

@scottelkin https://github.com/scottelkin I have been trying to reproduce the slowness you're seeing, but I have been entirely unsuccessful. I profiled the code and saw no unusual slowness when a post had dozens of tags with cache files to purge upon updating a post.

I also created a multisite test environment with dozens of child blogs, each with dozens of their own posts, and then pre-cached all of the posts in the entire network (2,700+ cache files). When I edited a post in a child blog, which resulted in Quick Cache purging the cache for any associated post terms (categories/tags), there was absolutely no slow-down or difference in purging speed (when compared to a single-site WordPress

install).

As I've been entirely unsuccessful reproducing the problem you're having, I have to assume that there's another variable at play here, something related to either your hosting environment or one of your other plugins introducing a conflict somewhere that is affecting Quick Cache.

If you can recreate this issue in a clean WordPress install (with no other plugins running) or provide me with any other details that might help me narrow this down, I will be happy to continue looking into this.

— Reply to this email directly or view it on GitHub https://github.com/websharks/quick-cache/issues/247#issuecomment-51431992 .

/ scott

[raamdev]

@jaswsinc Any thoughts on how APC might play into this issue?

[jaswrks]

Any thoughts on how APC might play into this issue?

I don't see APC having anything to do with this at all. If anything, APC would speed things up further. APC is a cache for code, it's not about the impact that certain routines would have at runtime.

I would focus my research on the underlying filesystem. These routines associated with purging are dealing with the filesystem. If you are seeing a larger impact (slowness), this could be attributed to slow disk I/O. Quick Cache is recursively scanning your filesystem to purge it of cache files.

@scottelkin Are you running an NFS? Or some other form of remote filesystem? Even if it's not "remote" is it actually on the same server or elsewhere in a network?

[scottelkin]

It is a virtual server setup at Softlayer (a division of IBM) with a local drive using the ext3 filesystem. (12 CPU "units" and 32 gigs of RAM)

On Thu, Aug 7, 2014 at 9:09 PM, JasWSInc notifications@github.com wrote:

Any thoughts on how APC might play into this issue?

I don't see APC having anything to do with this at all. If anything, APC would speed things up further. APC is a cache for code, it's not about the impact that certain routines would have at runtime.

I would focus my research on the underlying filesystem. These routines associated with purging are dealing with the filesystem. If you are seeing a larger impact (slowness), this could be attributed to slow disk I/O. Quick Cache is recursively scanning your filesystem to purge it of cache files.

@scottelkin https://github.com/scottelkin Are you running an NFS? Or some other form of remote filesystem? Even if it's not "remote" is it actually on the same server?

— Reply to this email directly or view it on GitHub https://github.com/websharks/quick-cache/issues/247#issuecomment-51561470 .

/ scott

[jaswrks]

@scottelkin Thanks! Does that have an SSD? Or, if it's a physical drive, how many RPMs?

Also, please report back the average size of your Quick Cache directory; i.e. if you scan the filesystem to determine how much data is inside of your /quick-cache/cache directory throughout the day (on average) what sort of numbers are we talking about?

[jaswrks]

@raamdev I think this issue underlines the need for us to have a closer look at trying to support an in-memory cache location; i.e. ElastiCache and other flavors of memcache. For extremely large sites that have the money to invest in such servers, this could provide an additional speed boost when it comes time for Quick Cache to deal with purging.

If the filesystem is quite large (i.e. /quick-cache/cache contains an extremely large number of files), this could impact performance. This is to be expected of course, but in these cases an in-memory cache could be a way to speed things up even further, even on very large sites.

It might also be worth us looking closer at the regex iteration routines in Quick Cache, to see if there is something more we can do to optimize them even further.

[jaswrks]

It might also be worth us looking closer at the regex iteration routines in Quick Cache, to see if there is something more we can do to optimize them even further.

We could look specifically at how Quick Cache deals with a MS network again, just to be absolutely sure that all of the purging routines are dealing only with the current sub-directory or sub-domain of the current blog. If there are any routines that might be looking at the entire network on-a-whole, and the network is HUGE, this could be an issue.

I'm not aware of that being the case at all in Quick Cache, but it couldn't hurt to have another look. The only functionality that I'm aware of (at this time), which looks at the entire network at once, would be the "wipe" functionality; where a network admin can wipe the entire cache for all blogs in the network.

[jaswrks]

@scottelkin When you log into the Dashboard of a child blog, and you look at the URL (i.e. the address bar) are you on a sub-directory installation, a sub-domain installation; or have you configured your network in some way that might move the Dashboard area for every blog to the same domain/path?

[scottelkin]

Each child blog is a 'sub-domain', but we use a plugin called "Domain Mapper" that allows them to map their domain name to the site. So a child blog even though their subdomain is set up at ' testsite.socialprofitmachine.com', they can map testsite.com to work correctly with it.

On Thu, Aug 7, 2014 at 9:45 PM, JasWSInc notifications@github.com wrote:

@scottelkin https://github.com/scottelkin When you log into the Dashboard of a child blog, and you look at the URL (i.e. the address bar) are you on a sub-directory installation, a sub-domain installation; or have you configured your network in some way that might move the Dashboard area for every blog to the same domain?

— Reply to this email directly or view it on GitHub https://github.com/websharks/quick-cache/issues/247#issuecomment-51562984 .

/ scott

[jaswrks]

Each child blog is a 'sub-domain', but we use a plugin called "Domain Mapper" that allows them to map their domain name to the site.

Perfect, that's perfectly compatible with Quick Cache then. ~ I'll wait to hear back about your average directory size.

[jaswrks]

@raamdev I think I found what could be causing the issue here. Notice lines like this one in the QC codebase. While the regex pattern limits the search to only files related to the current blog (and at first glance that would seem to be enough); however, the directory that we are scanning is still the primary /quick-cache/cache directory.

On a large MS network, this could require a lot more time than it needs to. We could work to improve this, so that we are only scanning the directories that we absolutely need to. Of course, there is more to it than that; but it's something we could take a look at.

[jaswrks]

@raamdev In this commit (while I was working on #182), I introduced a new class member and left you some notes. I think we could take the work from that issue and use it as a starting point for this one. Please see this commit where I left notes regarding delete_files_from_host_cache_dir()

[raamdev]

In this commit (while I was working on #182), I introduced a new class member and left you some notes. I think we could take the work from that issue and use it as a starting point for this one. Please see this commit where I left notes regarding delete_files_from_host_cache_dir()

Excellent. That sounds like a great idea. I'll mark this issue as a bug then and work towards consolidating those routines to improve overall performance with the expectation that doing so will address, at least partially, the problem described in this issue.

@scottelkin If you could get us some numbers about your average cache directory size, that would help us as well. :)

[scottelkin]

I'm sorry, I just don't know how I can best answer that question. Is there a specific command I can run? It is a WPMU installation with almost 400 sites, so there is a lot of directories and files. The total size of quick-cache/cache directory right now is 790M. Does that answer the question?

On Tue, Aug 12, 2014 at 9:23 PM, Raam Dev notifications@github.com wrote:

In this commit (while I was working on #182 https://github.com/websharks/quick-cache/issues/182), I introduced a new class member and left you some notes. I think we could take the work from that issue and use it as a starting point for this one. Please see this commit where I left notes regarding delete_files_from_host_cache_dir()

Excellent. That sounds like a great idea. I'll mark this issue as a bug then and work towards consolidating those routines to improve overall performance with the expectation that doing so will address, at least partially, the problem described in this issue.

@scottelkin https://github.com/scottelkin If you could get us some numbers about your average cache directory size, that would help us as well. :)

— Reply to this email directly or view it on GitHub https://github.com/websharks/quick-cache/issues/247#issuecomment-52008438 .

/ scott

[raamdev]

@scottelkin Knowing that the cache directory size is 790M is helpful. That's a pretty big cache!

Is there any chance you could get a count of how many files are in the cache directory?

If the server is Linux-based, and you have access to a command line, you could run the following command and share the output from that:

Count number of files in cache directory

$ find /path/to/wp-content/cache/quick-cache/ -type f | wc -l

Count number of directories in cache directory

$ find /path/to/wp-content/cache/quick-cache/ -type d | wc -l

Note: That's a lowercase L, not a 1.

[scottelkin]

Of just the quick-cache directory (not all the qc-* files in the parent cache directory):

files: 19498 dirs: 5259

On Mon, Aug 18, 2014 at 12:33 PM, Raam Dev notifications@github.com wrote:

@scottelkin https://github.com/scottelkin Knowing that the cache directory size is 790M is helpful. That's a pretty big cache!

Is there any chance you could get a count of how many files are in the cache directory?

If the server is Linux-based, and you have access to a command line, you could run the following command and share the output from that: Count number of files in cache directory

$ find /path/to/wp-content/cache/quick-cache/ -type f | wc -l

Count number of directories in cache directory

$ find /path/to/wp-content/cache/quick-cache/ -type d | wc -l

Note: That's a lowercase L, not a 1.

— Reply to this email directly or view it on GitHub https://github.com/websharks/quick-cache/issues/247#issuecomment-52542841 .

/ scott

[raamdev]

files: 19498 dirs: 5259

OK, thank you. Wow, that's a lot of cache files. :) I'll work on attempting to recreate a WPMU installation with 20k cache files. My latest test had 1.3k cache files.