Ability to move cache-clearing to WP cron or another async method

[archon810]

During my analysis for https://github.com/W3EDGE/w3-total-cache/issues/39 and https://github.com/W3EDGE/w3-total-cache/issues/35, I analyzed the time it takes for W3TC to clear cache (specifically for memcached), and even with our beefy setup, it still tacks on extra seconds (sometimes as many as 5-10s) to every Update/Publish button press, which wastes precious time for our writers (we're also still using the classic editor, which means post saves aren't done in AJAX).

A single post update could generate hundreds of cache gets and deletes, and that really adds up.

It'd be great to see a built-in option to switch cache-busting to WP cron so it would fire on the next minute and wouldn't waste valuable time for the writers. I envision this via a config parameter / variable, since we want to retain sync cache-busting for default scenarios.

@bwmarkle What do you think?

Thanks.

[bwmarkle]

Hi @archon810. I'm actually meeting with several of the other developers today, and I plan to discuss some of the GitHub issues that you've created. They're on the radar, and hopefully we'll be able to chat more on these issues soon.

Thanks! - Brad

[archon810]

Awesome, I'd love to be part of them in any way, as we have several large sites, have been using W3TC for years, and are running into some perf issues, which is ironic, given that it's supposed to improve performance, not make it worse.

Right now, we have an active exploration into migrating W3TC cache management operations on post updates to an async method. I personally don't like WP Cron, so we're exploring a task runner like RabbitMQ or Gearman with https://github.com/10up/WP-Minions, and then hijacking the W3TC jobs and giving them to the task queue for instant but async processing.

It's a bit convoluted, as we have to remove some hooks, so an upstream solution (like at least moving to WP Cron but also a possibility of easy integration with something like WP-Minions) would be great.

Another thing that could work out of the box is something like https://github.com/techcrunch/wp-async-task.

[bwmarkle]

Hi @archon810,

So, right now we're running through two different ideas.

IDEA 1

The first idea is that start off with a simple solution, and over time expand and improve upon it. That solution would be to offer users the option to NOT have cache purged whenever an update occurs. If they choose that option, they'd either have to use a WPCron to clear the cache or they would have to setup a cron job that simply calls a wp cli command to clear the cache.

When editing a page, authors could also have the option to clear the cache now and ignore the global settings.

One of the benefits of this idea is that it is relatively straight forward, and after the first release, we can iteratively improve it over time. Dev wise, it would not take much time to implement. The downside that we're worried about is that such a simple approach won't be sufficient for our more advanced users. We also don't want to release something so basic, and have the community think we don't really know what we're doing.

IDEA 2

Our second idea is more robust. We would use a queue system, but we would have an API service that more intelligently handles the purging.

For example, if you update 4 pages, the purge will clear the homepage 4 times, when it shouldn't need to.

We also think users will want the ability to set an order of importance to the purge process. For example, if you update a post that needs to be referenced on the homepage, the homepage should have a high priority of getting purged faster.

And... in addition to purging, there's also the idea of priming the cache. If we prime the cache ahead of time, it can help prevent scenarios where 100 people all access the homepage within the same second and cause 100 builds of the homepage. If we prime the cache ahead of time, the homepage cache will only be cleared once a new cached version is ready to take its place.

Anyways, I just wanted to share some of those thoughts to let you know we're thinking through this new feature. We're trying to weigh all the factors, especially simplicity vs efficiency, and we want to make sure we're working on something the community will really like.

Thanks,

• Brad

[archon810]

@bwmarkle IDEA 1 seems too simplistic, as it's not clear how the cron job would know what caches to dump. What would make more sense is setting a cron job from the cache clearing logic instead and passing the post ID in, though this has the danger of potentially creating thousands of jobs (if another process goes and updates a bunch of posts for whatever reason), and thousands of jobs will not scale in WP cron because it uses a single cron wp_options entry that gets overwritten over and over due to race conditions. See https://core.trac.wordpress.org/ticket/15148#comment:9.

IDEA 2 seems better, but it's unclear from the technical standpoint how it'd work.

On our end, we're very close to rolling out the solution I mentioned earlier - RabbitMQ and WP-Minions.

[archon810]

@maciejmackowiak Do you want to chime in with any improvements we'd want to make the above RabbitMQ/WP-Minions solution more streamlined or are all the necessary actions/filters already in place?

[maciejmackowiak]

I think that it would be perfect to:

1. Have settings in w3c to set cache update on post updated not immediately but in wp_schedule_single_event (we will not need this once we roll out RabbitMQ, but I believe it will be a step to the right direction). But like @archon810 mentioned it may be dangerous in some cases.
2. Have settings in w3c not to cache purged whenever an update occurs.
3. You will provide a function/action that we could use - same one that will be used for WP Cron cache purging.

This way I think You will cover both advanced and not advanced users.

BTW with current w3c version I was planning on using something like this:

if (\W3TC\Util_Environment::is_flushable_post(
        $post,
        'posts',
        \W3TC\Dispatcher::config()
    )){
        $cacheflush = \W3TC\Dispatcher::component('CacheFlush');
        $cacheflush->flush_post($post_id);

    }

Is it the correct approach?

[maciejmackowiak]

Any updates here?

[archon810]

@bwmarkle Would love for you to chime in here again and kick things off.

[cssjoe]

@archon810 and @maciejmackowiak Thank you for contributing thought to this issue. We are discussing options and want to make an update for this soon. One idea that we have is to add a new setting in wp-admin/admin.php?page=w3tc_pgcache#purge_policy, such as "Defer purging". There could be another setting for a purge interval for deferred cache items. We can add a new do_action() call to trigger the purging. We also can also add a WP-CLI command to trigger the action. Please let us know your thoughts.

[maciejmackowiak]

Hi @cssjoe

How about this - there could be 2 options - one to trigger purging via do_action/WP-CLI or a cron job that will update the posts.

For each publish/update/unpublished post you could mark a post with meta value as it needs purging.

When the purge action is triggered(cron job or wp-cli/do_action) it would purge/update those posts cache and also mark the corresponding categories/list pages that needs updating. After updating all the posts in the queue the categories/list pages etc would be updated. This way if 2 or more posts in the queue are corresponding to the same category/list page update of the same category it will be updated only once.

There is also a need for immediately updating cache for example when post goes from publish to not publish status, furthermore it would be nice to have some checkbox on post edit page that will allow user to force immediate purge of the post.

[cssjoe]

@maciejmackowiak Thanks for all of the feedback. I like the idea of a checkbox on a post page to purge immediately. This feature request is in our backlog and we'll provide an update when we get to work on it.

[cssjoe]

We are working on a purge queue in https://github.com/BoldGrid/w3-total-cache/pull/882 and plan to release the feature soon. The new feature should address the concerns in this issue.