search

putyourlightson / craft-blitz

Intelligent static page caching for creating lightning-fast sites with Craft CMS.
https://putyourlightson.com/plugins/blitz
Other
147 stars 35 forks source link

Cloudflare reverse proxy purge not working #661

Closed davidwebca closed 2 weeks ago

davidwebca commented 2 months ago

Bug Report

Hi,

I have configured Blitz first and everything's working fine and now since I added Cloudflare in front with reverse proxy purging, I don't see the content update ever. When updating a page, I was expecting Cloudflare to be purging the URL of the entry updated and all its related entries (trackElements and trackElementQueries are both set to true).

Diagnostics Report

Application Info

Installed Plugins

Loaded Modules

Blitz Plugin Settings

{
    "debug": false,
    "hintsEnabled": true,
    "cachingEnabled": true,
    "refreshCacheEnabled": true,
    "refreshMode": 3,
    "includedUriPatterns": [
        {
            "siteId": "",
            "uriPattern": "*"
        }
    ],
    "excludedUriPatterns": [
        {
            "siteId": "",
            "uriPattern": "caedfb0c-010c-4f8a-a371-c8dedf9fcbe2"
        }
    ],
    "cacheStorageType": "putyourlightson\\blitz\\drivers\\storage\\FileStorage",
    "cacheStorageSettings": {
        "folderPath": "@webroot\/cache\/blitz",
        "compressCachedValues": ""
    },
    "cacheStorageTypes": [],
    "cacheGeneratorType": "putyourlightson\\blitz\\drivers\\generators\\LocalGenerator",
    "cacheGeneratorSettings": {
        "concurrency": "5"
    },
    "cacheGeneratorTypes": [],
    "customSiteUris": [],
    "cachePurgerType": "putyourlightson\\blitz\\drivers\\purgers\\CloudflarePurger",
    "cachePurgerSettings": {
        "zoneIds": {
            "5f474987-e317-4e5e-8177-1021a44afefc": {
                "zoneId": "9ede78a1cc4bad23789b5d9aa4d9b4f9"
            },
            "437c9636-97d8-4461-805c-b7aae67cadac": {
                "zoneId": "9ede78a1cc4bad23789b5d9aa4d9b4f9"
            },
            "bd2d0bfb-89c8-4f7a-bf21-89928ee29374": {
                "zoneId": "9ede78a1cc4bad23789b5d9aa4d9b4f9"
            }
        },
        "authenticationMethod": "apiKey",
        "apiToken": "****************************************",
        "apiKey": "*************************************",
        "email": "web@rent-a-boxer.com"
    },
    "cachePurgerTypes": [],
    "deployerType": "putyourlightson\\blitz\\drivers\\deployers\\DummyDeployer",
    "deployerSettings": [],
    "deployerTypes": [],
    "ssiEnabled": false,
    "ssiTagFormat": "<!--#include virtual=\"{uri}\" -->",
    "detectSsiEnabled": true,
    "esiEnabled": false,
    "queryStringCaching": 0,
    "includedQueryStringParams": [
        {
            "siteId": "",
            "queryStringParam": ".*"
        }
    ],
    "excludedQueryStringParams": [
        {
            "siteId": "",
            "queryStringParam": "gclid"
        },
        {
            "siteId": "",
            "queryStringParam": "fbclid"
        }
    ],
    "apiKey": "",
    "generatePagesWithQueryStringParams": true,
    "purgeAssetImagesWhenChanged": true,
    "refreshCacheAutomaticallyForGlobals": true,
    "refreshCacheWhenElementMovedInStructure": true,
    "refreshCacheWhenElementSavedUnchanged": true,
    "refreshCacheWhenElementSavedNotLive": false,
    "cacheNonHtmlResponses": false,
    "trackElements": true,
    "trackElementQueries": true,
    "excludedTrackedElementQueryParams": [],
    "cacheElements": true,
    "cacheElementQueries": true,
    "cacheDuration": null,
    "nonCacheableElementTypes": [],
    "sourceIdAttributes": [],
    "liveStatuses": [],
    "integrations": [
        "putyourlightson\\blitz\\drivers\\integrations\\CommerceIntegration",
        "putyourlightson\\blitz\\drivers\\integrations\\FeedMeIntegration",
        "putyourlightson\\blitz\\drivers\\integrations\\SeomaticIntegration"
    ],
    "defaultCacheControlHeader": "no-cache, no-store, must-revalidate",
    "cacheControlHeader": "public, s-maxage=31536000, max-age=0",
    "cacheControlHeaderExpired": "public, s-maxage=5, max-age=0",
    "sendPoweredByHeader": true,
    "outputComments": true,
    "refreshCacheJobPriority": 10,
    "driverJobBatchSize": 100,
    "driverJobPriority": 100,
    "queueJobTtr": 300,
    "maxRetryAttempts": 10,
    "maxUriLength": 255,
    "mutexTimeout": 1,
    "commands": [],
    "injectScriptEvent": "DOMContentLoaded",
    "injectScriptPosition": 3
}

Recommendations

Site Tracking – EN

Site Tracking – DE

bencroker commented 2 months ago

Your configuration looks fine. Do you see any error messages when saving the Blitz plugin settings from the Cloudflare purger? And have you followed the instructions for enabling static page caching in Cloudflare at https://putyourlightson.com/plugins/blitz#cloudflare-page-rules?

davidwebca commented 2 months ago

Yes, I followed everything for static page caching and it all works, I've just re-verified just to be sure, and no I don't see any error. To me, it seems like Blitz's cache is being refreshed correctly when saving an element. If I put Cloudflare on "dev mode" temporarily and save an element, everything works fine but not Cloudflare.

One thing that might be specific to my setup is that my host doesn't have a cron job editor so I have setup a remote call to my cron job (excluded by the cache of course) that runs the Craft queue. I've recreated the "run queue" call with some modifications and included Blitz's refreshExpiredCache method call in there so that they get added to the queue right before running it. 

<?php 
namespace rab\controllers;

use Craft;
use craft\helpers\App;
use craft\helpers\Json;
use craft\elements\Entry;
use craft\web\Controller;
use craft\queue\QueueInterface;
use yii\web\Response;
use putyourlightson\blitz\Blitz;

class MainController extends Controller
{
    protected array|bool|int $allowAnonymous = ['run'];
    private QueueInterface $queue;

    public function init(): void {

        parent::init();
        $this->queue = Craft::$app->getQueue();
    }

    public function actionAll() {
        // return $this->getView()->render();
    }

    public function actionRun() {
        // Prep the response
        $this->response->content = '1';

        // Make sure the queue isn't already running, and there are waiting jobs
        if ($this->queue->getHasReservedJobs() || !$this->queue->getHasWaitingJobs()) {
            return $this->response;
        }

        // Cron job refreshes expired cache
        Blitz::$plugin->refreshCache->refreshExpiredCache();

        // Run the queue
        App::maxPowerCaptain();

        $this->queue->run();

        return $this->response;
    }
}

Maybe that's the source of the problem? Should I be adding something specific to Cloudflare in there?

davidwebca commented 2 months ago

I don't know the code base really well and I'm probably wrong but it seems to me that the refreshExpiredCache doesn't call the cachePurger, so maybe I should create a queue job that runs right after and calls something like

Blitz::$plugin->cachePurger->purgeUris($purgeableSiteUris);

?

bencroker commented 2 months ago

Your code looks fine to me and shouldn’t be causing any issues with the cache purger.

Are you seeing any errors in the Blitz log file at storage/logs/blitz-*.log?

davidwebca commented 2 months ago

No errors and in the queue logs I also get this.

2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117837] Refreshing Blitz cache (attempt: 1, pid: 27618) - Started {"memory":5449216} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::afterExec]  [117837] Refreshing Blitz cache (attempt: 1, pid: 27618) - Done (time: 0.014s) {"memory":5557688} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117838] Refreshing Blitz cache (attempt: 1, pid: 27618) - Started {"memory":5565608} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::afterExec]  [117838] Refreshing Blitz cache (attempt: 1, pid: 27618) - Done (time: 0.005s) {"memory":5586192} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117846] Refreshing Blitz cache (attempt: 1, pid: 27618) - Started {"memory":5594112} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::afterExec]  [117846] Refreshing Blitz cache (attempt: 1, pid: 27618) - Done (time: 0.005s) {"memory":5614696} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117847] Purging Blitz cache (attempt: 1, pid: 27618) - Started {"memory":5634968} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::afterExec]  [117847] Purging Blitz cache (attempt: 1, pid: 27618) - Done (time: 0.982s) {"memory":5657776} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117848] Generating Blitz cache (attempt: 1, pid: 27618) - Started {"memory":5667240} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::afterExec]  [117848] Generating Blitz cache (attempt: 1, pid: 27618) - Done (time: 0.007s) {"memory":5528624} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117849] Purging Blitz cache (attempt: 1, pid: 27618) - Started {"memory":5552936} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::afterExec]  [117849] Purging Blitz cache (attempt: 1, pid: 27618) - Done (time: 0.689s) {"memory":5568600} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117850] Generating Blitz cache (attempt: 1, pid: 27618) - Started {"memory":5577096} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::afterExec]  [117850] Generating Blitz cache (attempt: 1, pid: 27618) - Done (time: 0.002s) {"memory":5625728} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117851] Purging Blitz cache (attempt: 1, pid: 27618) - Started {"memory":5650096} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::afterExec]  [117851] Purging Blitz cache (attempt: 1, pid: 27618) - Done (time: 0.807s) {"memory":5665888} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117852] Generating Blitz cache (attempt: 1, pid: 27618) - Started {"memory":5683544} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::afterExec]  [117852] Generating Blitz cache (attempt: 1, pid: 27618) - Done (time: 0.002s) {"memory":5732176} 
2024-05-04 20:45:33 [queue.INFO] [craft\queue\QueueLogBehavior::beforeExec]  [117839] Generating motorcycle sitemap (attempt: 1, pid: 27618) - Started {"memory":5824176}
bencroker commented 2 months ago

So purging the cache appears to be working. Do you see the following messages in the Blitz log file and are they what you would expect to see?

[2024-04-30 03:04:14] 10 page(s) purged.
davidwebca commented 1 month ago

After verification, the cron job doesn't log anything in Blitz logs, even when trying to force the 'monologTargetConfig''s level to Psr\Log\LogLevel::DEBUG. Is the call to Blitz::$plugin->refreshCache->refreshExpiredCache() not enough? 🤔

davidwebca commented 1 month ago

I've continued to dig in since I have time to spend on this project this afternoon and I've found that adding Blitz::$plugin->cachePurger->purgeAll() right before Blitz::$plugin->refreshCache->refreshExpiredCache() does the trick and purges Cloudflare and Blitz correctly. I'd like to dig in more to actually only purge what is necessary and not "all". 🤔

bencroker commented 1 month ago

Since you’re using the refresh mode Clear the cache and regenerate in a queue job, modifying any entry should result in pages that output its fields to be regenerated and purged immediately. Calling refreshExpiredCache() is only relevant in this case for entries with future post or expiry dates.

I noticed some possible issues with your URI patterns.

    "includedUriPatterns": [
        {
            "siteId": "",
            "uriPattern": "*"
        }
    ],
    "excludedUriPatterns": [
        {
            "siteId": "",
            "uriPattern": "caedfb0c-010c-4f8a-a371-c8dedf9fcbe2"
        }
    ],

Please change "*" to ".*", since this is a regular expression.

Also, is the URI pattern caedfb0c-010c-4f8a-a371-c8dedf9fcbe2 intentional?


    "cachePurgerSettings": {
        "zoneIds": {
            "5f474987-e317-4e5e-8177-1021a44afefc": {
                "zoneId": "9ede78a1cc4bad23789b5d9aa4d9b4f9"
            },
            "437c9636-97d8-4461-805c-b7aae67cadac": {
                "zoneId": "9ede78a1cc4bad23789b5d9aa4d9b4f9"
            },
            "bd2d0bfb-89c8-4f7a-bf21-89928ee29374": {
                "zoneId": "9ede78a1cc4bad23789b5d9aa4d9b4f9"
            }
        },

Each of the 3 sites share the same zone ID. Is that because they all share the same subdomain?

Everything else looks fine, apart from the fact that the number of pages purged is not being logged in blitz-*.log.

bencroker commented 1 month ago

Any update on this, @davidwebca?

davidwebca commented 2 weeks ago

Hi! Sorry for the delay responding. This has caused much of a headache and we had weird interactions between Cloudflare's cache and Google Tag Manager / Google Ads retargeting (it was injecting stuff in forms' action URL to enable a "privacy-friendly" way to track, but Cloudflare sometimes wouldn't purge it depending on the region, even when going on their admin interface).

In the end, we had to remove Cloudflare altogether which is sad because I was also using their GeoIP rules to set the default website depending on the location of the visitor.

So to answer your question: the three websites are 3 different languages on the same domain in Craft, so that's why the zoneId is the same.

In the end though, regarding this current issue, it seems like the reverse purge doesn't work on save and has to be called again later, so my idea of the queue task in a cron job in the previous comments was working. I don't know if it's something wrong with the order that the reverse proxy purge code runs or if it's something with my setup - OPCache maybe? - but that's how I had solved it in the end before pulling the plug on Cloudflare.

bencroker commented 2 weeks ago

This issue was very likely specific to your setup as I haven’t had any such reports. If you ever come back to using Cloudflare for this site and want to dig in further then just let me know.