Question: Performance when warming cache for 500+ entries, a lot of which redirect to others

[jannisborgers]

Cache warming takes super long

We’re maintaining a site with a handful of structure sections, each is nested 3-6 levels deep. Entries in some of the structure levels are only saving the data, but instead of outputting anything when you visit their URL, they redirect to their parent entry, which shows their content next to their siblings’ content.

While this is great for DRY content editing, it leads to a system with 500+ entries, with a significant portion just redirecting to their respective parents in the template code.

Cache warming takes a couple of hours for this site in local dev and on our staging server, and I could easily fry pancakes on my development Macbook. I’m not sure what part of the setup is the bottleneck here. I was expecting that I would have to write some pretty complex RegEx to either include or exclude URLs based on whether they’re just redirecting to their parent (don't cache) or displaying content (cache). My thinking was that the warmer first checks the URI patterns and we can skip all the requests to the redirecting entry URLs.

Site performance the problem?

But the more I think about it, it seems that the overall site performance might contribute the bulk of the long cache warming, as it takes a couple of seconds to load each entry. We’re talking about the queue counter going up every 15-25 seconds or so. Having removed the {% cache %} tags that were in place before, performance without the pages being in the Blitz cache has decreased immensely, of course.

Additional context

We’re running Craft 3.6.17 with Blitz 3.11.1 on nginx (Nitro, locally) and Apache (Staging), respectively.

[bencroker]

First of all, which warmer driver are you using? In my experience, the Guzzle Warmer can warm up to 5 pages per second on a performant site, so something about your setup doesn't sound right.

Guzzle follows redirects by default (https://docs.guzzlephp.org/en/stable/quickstart.html#redirects), but you can tell it not to, which might help you, by adding the following to config/guzzle.php.

return [
    'allow_redirects' => false,
];

[jannisborgers]

Hi @bencroker, thanks for the quick answer!

I’m using Guzzle as it’s recommended.

Setting allow_redirects to false resulted in the caching queue job to run in just 5 seconds. Debugging showed cURL error 7 for all entries.

I then tried using the nitro craft blitz/cache/warm console command, which didn’t work either, then I tried ./craft blitz/cache/warm and suddenly the speed you described (3-5 entries per second) was reached.

I had to set "generateTransformsBeforePageLoad" => true as per the docs because most entries weren’t cached due to transform generation URLs.

Now the speed is fine, it ran through all entries in a couple of minutes! 😊

Not sure why running Blitz inside the Nitro container didn’t work, though. I suppose I’d need to point to 0.0.0.0 instead of my site URL in order to get it to work… (?)

[bencroker]

Not sure why running Blitz inside the Nitro container didn’t work, though. I suppose I’d need to point to 0.0.0.0 instead of my site URL in order to get it to work… (?)

Yeah, very likely a routing issue. I rarely test Blitz warming locally though, I find it much more reliable (and accurate) to test in a staging environment.