Feature requests: auto-initialize cache for new posts / crawl sitemap

[seyfro]

First thanks for your great plugin - we first used another caching solution for a high-traffic website but unfortunately none really worked as we needed it to. Nevertheless there is still one issue I hope you can help with (by adding a new feature ;-)

Currently if you publish a post, the related html file only gets created if the URL is called once by a non-logged in user. On our site (where an additional htcache is used in front of Wordpress) this lead to database timeouts, as we experienced a peak immediately after the post was published (and as there was not enough time to initialize the htcache based on the html file created by cache enabler).

In the best case, the risk for this to happen again could be reduced by adding two features/option to the plugin:

1. add an option to initialize a single cache html file immediately after a post is published

2. add an option to initialize the full cache for all content if a sitemap url is given (in this case, the plugin could trigger a job to crawl all URLs from the sitemap so that the cache gets prefilled before a real person hits the page)

I think these feature could be helpful to other high-traffic sites too.

If this is not possible, is there perhaps a best practice on how to best automatically initialize the cache without waiting for the first non-logged-in user?

[seyfro]

FYI - we built some snippets to trigger the auto-initialisation based on a sitemap, I am sharing it here in case someone is interested:

snippet for functions.php:

/*
initialize cached enabler files after publication, post update & sending to trash
*/
add_action('init', 'customprefix_register_hooks', 198);
function customprefix_register_hooks() {
    $post_types = get_post_types(
        array('public' => true)
    );
    if (empty($post_types)) {
        return;
    }
    foreach ($post_types as $post_type) {
        add_action("publish_{$post_type}", 'customprefix_rebuild_cache', 20, 2);
        add_action("post_updated_{$post_type}", 'customprefix_rebuild_cache', 21, 2 );
        add_action("trashed_post", 'customprefix_rebuild_cache', 22, 2 );
    }
}
function customprefix_rebuild_cache() {
    exec('php /rebuild-cache.php > /dev/null &');
}

rebuild-cache.php:

<?php
/* 
initialized cache enabler cacher based on sitemap (created by XML Sitemap & Google News feeds plugin)
triggered by publish & post_updated hooks in functions.php
*/
file_put_contents('/rebuild-cache.running', date('Y-m-d H:i:s'), FILE_APPEND);
$curl = curl_init();
$url = 'https://www.domain.com/sitemap.xml';
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_HEADER, false);
$data = curl_exec($curl);
$index = new SimpleXMLElement($data);
foreach ($index->sitemap as $sitemap) {
    $sitemap_urls = $sitemap->loc;
    curl_setopt($curl, CURLOPT_URL, $sitemap_urls);
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl, CURLOPT_HEADER, false);
    $subdata = curl_exec($curl);
    $subindexes = new SimpleXMLElement($subdata);
    foreach ($subindexes->url as $url) {
        curl_setopt($curl, CURLOPT_URL, $url->loc);
        curl_setopt($curl, CURLOPT_HEADER, 0);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
        $data = curl_exec($curl);
    }
}
curl_close($curl);
unlink('/rebuild-cache.running');
echo 'done';

[seyfro]

Just one request: would it be possible to add a hook which gets fired once cache is cleared, like adding do_action("ce_action_cachepurged"); to clear_total_cache() function in cache_enabler.class.php once the cache is cleared? This would allow us to also use add_action("ce_action_cachepurged", 'customprefix_rebuild_cache', 23, 2 );

[svenba]

The action hook ce_action_cache_cleared has been added https://github.com/keycdn/cache-enabler/commit/4dc75a803c320f729e28dc9401c9cde3aa95b219.

[seyfro]

thanks for adding this hook!

As we also made some modifications to the CE plugin in the meantime, I wonder if it would also be possible to add hooks for the clear functions clear_page_cache_by_post_id(), clear_page_cache_by_url() and clear_home_page_cache()? This would give the maximum flexibility in rebuilding the cache automatically without having to change the plugin files.

[svenba]

We've added more action hooks to cover this: https://github.com/keycdn/cache-enabler/commit/640f3c4473d53d44e16cced6039fd2bf57ac9325

[seyfro]

thanks!