Cloudflare Page Cache Plugin for WordPress does not cache properly with W3TC

[BradFD]

When the very popular W3 Total Cache's page cache (disk enhanced mode) is enabled a very common setting, the Cloudflare Page Cache Plugin header "x-HTML-Edge-Cache: cache,bypass-cookies=wp-|wordpress|comment|woocommerce " is not added to the disk cached page. This results in the requests not being cached at the worker level since they don't see the proper header. I verified this by turning off W3TC page cache and then seeing immediate caching and hits at worker the worker level whereas before nothing really appeared to be caching.

[ScottTravisHartley]

Could you try, this? In the CloudFlare Page Cache Plugin around line 48, you will see a line like the one below.

add_action( 'init', 'cloudflare_page_cache_init_action'); Could you try replacing it with something like this? add_action( 'init', 'cloudflare_page_cache_init_action', PHP_INT_MAX );

[pmeenan]

Did you purge the W3TC cache after installing the Cloudflare page cache plugin? If not it is possible that an older version of the page is being served directly from disk (sometimes before it gets to php if it is using the Apache cache). I use W3TC on my test domain as well and it worked last time I checked but the cache does need to be cleared the first time.

Usually W3TC puts a banner across the top of the admin page when you install a new plugin recommending that the local cache be purged.

[BradFD]

@ScottTravisHartley Does not seem to make a difference will test some more.

@pmeenan Multiple full W3TC purges since first installed a week or so ago.

So if I put Cloudflare in Development Mode and clear a page from the W3TC Cache and then reload it in not logged in browser, I can see the header "x-HTML-Edge-Cache: cache,bypass-cookies=wp-|wordpress|comment|woocommerce" on the FIRST load of that page. However, when I hit F5 to reload that page again the header the cloudflare page cache plugin adds is gone. Could this be because the first time it is loading it is not in the W3TC page cache but the second time it is and the edge cache header has been stripped.....?

[ScottTravisHartley]

In the browser cache setting do you have the remove cookies option enabled? Also does it do the same when disk cache is set to basic as opposed to enhance?

[BradFD]

Remove cookies is not enabled in browser cache. So more testing, Cloudflare Development Mode on:

Test 1: Switching to Page Cache Disk: Basic, no change

Test 2: Turn W3TC Page Cache Off, and I see see "x-HTML-Edge-Cache: cache,bypass-cookies=wp-|wordpress|comment|woocommerce" everywhere on every page load.

Test 3: Turn W3TC Page Cache back ON, first F5 reload, I see header. All subsequent reloads no header from the Cloudflare plugin.

So it is really looks like the Page Cache of W3TC is stripping the header out....

[pmeenan]

My guess is that the page cache is serving the HTML without ever hitting php and the headers can't get added. I have some thoughts on how to deal with it that should solve the problem (will implement them today):

Plugin - Change the plugin to always add the header. For logged-in users it would explicitly say "nocache". That way a missing header means either the plugin isn't installed or the page was a cached page served directly.

Worker - Change the worker to have a server-side list of default cookie prefixes to bypass and do a stale-while-revalidate for HTML that is in cache but without the header set (and when the request doesn't have one of the default cookie prefixes). That means it will serve the cached version of the HTML and asynchronously go back to the origin to update the cache.

In the normal case where the plugin is installed, page edits will still trigger a purge and there will be minimal downside (just no reduction in number of requests to the origin but they will be background requests for cache updates). If the plugin isn't installed then a stale copy of the page will be served once while the cache is updated.

If a user doesn't have the plugin installed AND updated the login cookie to use a different prefix then things could go sideways with cached copies of admin pages being saved.

[pmeenan]

The 3 commits from earlier today should enable the logic I described above (the plugin has been updated on wordpress.org as well so it should be available as an update).

Basically, if the headers aren't there then the edge will serve a stale copy and update the edge cache from the origin on every page view. If the plugin is enabled it should still purge the cache on any edits preventing stale content from ever being served.

[BradFD]

Thank you. I really appreciate it. I will try to test this evening on a site with good traffic.

[BradFD]

Only seeing a status of "Miss" after updating the plugin, worker script, and clearing.

[pmeenan]

OK, that's bizarre. Does it say "Miss, Refreshed" or just "Miss"? It shouldn't be possible to just straight-up miss but it might be tripping over something I didn't handle.

Any chance you can share a URL where it is live (privately to pmeenan@webpagetest.org if you don't want the URL public)?

[BradFD]

Working thanks! On an installed WordPress site, I am seeing the TTFB audit in Google Pagespeed report under 100ms for the root document many times!

[wpsumo]

@pmeenan Lately I see a very low amount of Hit ratio and cached % in the Cloudflare dashboard. I sue the workers KV and it's very low. On some applications, it's even as low as 5%. And for some 20-40% which is very low. Previously I could see a ratio of 97-99%. Have you checked the compatibility lately? Wordpress 5.2?

See here: https://tppr.me/Cc6Vc https://tppr.me/q9bsS

Never experienced this with the API and normal worker, and never in the first month of the KV.

@ScottTravisHartley and using WP rocket and the latest version of the streamer in workers KV, the HTML snippet plugin and Wordpress 5.2.

See bypass all the time when I check in dev tools

[pmeenan]

Here is my test blog on 5.2 and the HTML is being cached (and the analytics show a high hit rate).

If you check in dev tools, make sure to use an incogneto tab (to make sure there are no cookies) and uncheck the "bypass cache" check box in dev tools. Or you can run some WebPageTest tests back-to-back (to give the first round a chance to prime the cache) and check for the x-HTML-Edge-Cache response headers.

I have seen multiple reports of the same issue so something is definitely going on. Maybe try going into the workers editor, edit the script and resources without making any changes but re-saving the current settings. I'm wondering if something in the path "forgot" about the KV config as they get ready to take KV out of beta.

[wpsumo]

I checked in the incognito and disabled cache and see the Hit and version now. Might be I missed tick the disable cache or incognito and see the response headers. https://tppr.me/GFyDw

But is there a conflict between the workers and the cache ratio dashboard in cloudflare? Since as you can see the significant drop it even looks like not even the static assets are delivered from the cache. And for the one with like 1-2% cache ratio we have not touched the site so it should trigger a flush every hour for it to be that low.

But none of the sites deliver a 90%+ anymore. I will try to open the workers script editor tomorrow and resave it. And let you know during the week if I see any improvements.

[kentonv]

We recently changed -- fixed, really -- the way cache hit ratio is reported in the dashboard. Previously, all workers requests were incorrectly interpreted as cache hits, which was clearly wrong. Now, we decide cache hits based on the CF-Cache-Status header on the response. If your worker passes through this header, then it should be counted correctly. If you strip this header off then you'll see 0% cache hit rate.

[wpsumo]

@kentonv Many thanks for sharing the information. But it still does not make sense why we have 1-2% cache hits. All static should even if the worker doesn't work be cache and normal rates to be on 30-40% as per default in cloudflare. So how come we see such low numbers?

@pmeenan Is better to answer if the worker passes through this header.

[pmeenan]

Yes, the worker passes through the header unmodified for most requests and for cases where the response is pulled from the worker cache it explicitly makes sure it is set to CF-Cache-Status: HIT. You should be able to verify the headers in the responses in dev tools and WebPageTest.

[pmeenan]

btw, if you want to try just passing the response through without explicitly setting it to HIT you can comment out this line: https://github.com/pmeenan/cf-workers/blob/master/streaming-optimizations/streaming-optimizations.js#L313

[wpsumo]

Yes it HIT but I see so low % cached resources in the cloudflare dashboard which is very strange don't you agree?

Thanks but made no changes to the cache hit ratio in the cloudflare dashboard

x-content-type-options: nosniff
x-html-edge-cache: cache,bypass-cookies=wp-|wordpress|comment_|woocommerce_
x-html-edge-cache-status: Miss, Cached
x-html-edge-cache-version: 112
x-kinsta-cache: MISS

And then browse and brows back

x-content-type-options: nosniff
x-html-edge-cache: cache,bypass-cookies=wp-|wordpress|comment_|woocommerce_
x-html-edge-cache-status: Hit
x-html-edge-cache-version: 112
x-kinsta-cache: MISS

This is what I still see in my dashboard: https://tppr.me/CqWWz https://tppr.me/faOMq

Looks like it flushes every time while we have not manually flushed or triggered anything that would trigger flush, no comment functions on the site. Do not update plugins/wp/theme every day or hour.

Worker: https://tppr.me/VsrwL

[pmeenan]

Is the cache version increasing each time (currently 112)? That will let you know if it is explicitly flushing or if the problem is that it is being aged out of the Cloudflare edge cache prematurely.

[wpsumo]

each time I fetch the url? or if it resets if I flush the edge cache?

[pmeenan]

Without manually flushing, Next time you see x-html-edge-cache-status: Miss, Cached see if the version number also updated x-html-edge-cache-version: 112.

My guess is not, otherwise it would be a LOT higher than 112 at this point. The version number is how the page cache explicitly purges itself when content changes. If you see a cache miss after a while but with the same version number it is because it was evicted from cache by Cloudflare (not requested frequently or cache pressure), not because it was purged. If that's the case it may be worth opening a ticket with support to see why your content is being evicted.

[pmeenan]

btw, it's worth checking the CF-Cache-Status: header as well to make sure it is being populated since that's how the cache hit rate is calculated.

[wpsumo]

Ok, I understand let's check next week and see what numbers I see. I can trigger a save on the page to trigger an edge-cache flush next week as well to see.

Yes CF-Cache-Status: is HIT as well.

But you mean I should not worry about the low cache ratio in the Cloudflare dashboard?

[pmeenan]

Not saying not to worry, for sure. Just trying to get to the root cause of the low hit rate. My guess is that the worker is caching the page and using the cache but that the resources aren't actually kept in the Cloudflare edge cache for as long as requested (the worker caches everything for a year). My guess is that unless the resources are hit pretty frequently that Cloudflare is evicting them automatically.

[wpsumo]

Got it! Ok, but why would Cloudflare evicting them and for so short of time? You mean the resources are fetched and the traffic is too low? But Cloudflare cache headers are set to respect the origin server as well and as you say. The HTML edge cache is held for 1 year.

[pmeenan]

If that ends up being the case then a Support ticket is probably the best way to prod for answers.

[wpsumo]

At cloudflare? Already have 5days without any reply. Just the automated answer based on the category you select of the ticket.

[wpsumo]

@pmeenan Can it be an issue by them sharing the same workers KV ID? I can not assign different ID's per site sharing the same name? Which potentially should be the right thing to do to not share the same and have individual containers.

And I don't know if this works as intended since I see this on the fresh new launched worker on a site. Since they share the same KW ID.

x-html-edge-cache-status: Hit, Refreshed
x-html-edge-cache-version: 112

And update of cache ratio. Still the same issue: https://tppr.me/dcmlu

[wpsumo]

@pmeenan Answer I got:

Hi there,

I would like to clarify a few things. The analytics app in the Cloudflare Panel is what we call HTTP Analytics and shows cache ratio for visitor <--> Cloudflare Cache.

This shows the Cache ratio between requests served from Cloudflare Cache and Origin (uncached). Up until the 8th May, this marked all requests to workers as cached, which was incorrect. From the 8th May, Cloudflare marked all requests to workers as uncached. This is because the initial request from visitor <--> Cloudflare isn't directly returned from cache, an additional request is made internally through your workers script. As this request requires additional processing, this counts as uncached. So, in terms of HTTP analytics, this counts as uncached but this doesn't mean the asset was served from origin, just that the worker returned the asset.

To find out whether the asset was served from Cache, you will need to look at Worker analytics, where I can see its around 30% cache hit. Here we are looking at the responses from the work subrequests, to see whether it was served from Cloudflare (cached) or origin (uncached).

As you are using Workers to cache everything (an approach which is not that common), you should ignore the HTTP analytics and focus only on Worker analytics.

Going back to the HTTP analytics, the reason you're seeing a small % of cached hits, is that these are not going through workers but our traditional caching flow.

I hope the above make sense and clarifies the issue.

[wpsumo]

@pmeenan See the logs from cloudflare below which tells me the worker is not working as intended.

I have checked our logs (1% sample data) for the last 7 days and these are the percentages of cache vs non-cached requests.

┌──requestPercentage─┬─cacheStatus─┐
│  70.48503611971104 │ unknown     │
│  6.191950464396285 │ miss        │
│  3.199174406604747 │ expired     │
│  19.09184726522188 │ hit         │
│ 1.0319917440660475 │ revalidated │
└────────────────────┴─────────────┘

unknown - means that 70% of the requests are being served from your origin.

[kentonv]

@alriksson I think that "unknown" implies that Cloudflare did not consider the URLs to be cacheable. By default Cloudflare will only attempt to cache URLs that end in certain file extensions, but you can change this behavior with settings. See:

https://support.cloudflare.com/hc/en-us/articles/200172516-Which-file-extensions-does-Cloudflare-cache-for-static-content-

https://support.cloudflare.com/hc/en-us/articles/115000150272-How-do-I-use-Cache-Everything-with-Cloudflare-

Unfortunately @pmeenan is no longer at Cloudflare and I don't know anything about this worker, the plugin, nor Wordpress in general, so I don't have any particular advice I can give here.

[wpsumo]

@kentonv Yes but with this worker we would cache HTML and host other things locally and in the egde such a google fonts etc. So if it was without worker yes it would sound OKish but not with the worker. And we already do cache everything and HTML but with the worker so it can bypass admin and logged in. Since without enterprise or business page rules is not suitable since it still cache if you´re logged in or in a /wp-admin/ path which causes issues. But even then this worker solve many other optimizations which is great. So I hope this development from Patrick or cloudflare does not stop.

Ok, but who owns this worker, @pmeenan or cloudflare?

[pmeenan]

I still don't think the response from support is accurate. I was under the impression that browser <-> Worker responses would be reported based on the CF-Status response header and if the workers return a response that was pulled from cache that it would correctly report in the top-level analytics as a cache hit.

If that was not the case then any site with a worker route that routes all traffic through the worker would show a zero % cache hit rate.

For example, this is the route configured for my WordPress test site:

I just ran a bunch of test traffic to it from WebPageTest and verified that I'm seeing CF-Cache-Status: HIT in the responses. 100% of the requests are being routed through the worker (in this case, the one that includes the HTML caching as well as font optimizations, etc). I'm seeing close to a 100% cache hit rate which is what you'd expect:

If it was behaving as support reported then it would be at 0% since every request is routed through a worker.

It is possible that you need a page rule to put the Cloudflare cache into "standards mode" where it checks the cacheability of every request and honors the cache-control headers that are set:

That said, if you are seeing CF-Cache-Status: HIT in the response headers then I'm assuming it is checking cache correctly already which is what made me think that the requests were aging out of cache.

Are there multiple domains covered by the same zone, maybe the misses are coming from other content than the HTML?

As far as ownership goes, that's kind of the point of it being an example and being open-source. Cloudflare owns the wordpress.com-distributed version of the plugin in the plugin gallery (because somebody has to) but the worker code is free for anybody to work on and contribute to. I'm happy to help understand what is going on and fix any issues that come up with the worker itself.

The behavior of the underlying cache and reporting UI though are outside of my control.

[wpsumo]

@pmeenan many thanks for the time you take to answer and help out!

Yes I try to understand what the support says and how to read the analytics dashboard as well if they would be right. I have a hard time to understand where I would see my cache ratio and why it would be like they said. And as you confirm and can see this is what I saw before but saw this significant drop which indicated something is wrong.

Regards the page rules, this should not be needed upon original to use the cache level > cache everything? You just suggested this and the origin cache control > on as an alternative solution, correct? It's super strange since everything used to work good and I saw what you just shared around 90-95% cache hit ratio.

We have e.g 1 account with 3 applications sharing the same zone since that's the way it should be set up no possibility to isolate them to unique workers? What I see is a better ratio of sites that have their own dedicated account and its own worker and workers kV. But it's different sites and projects but even they see a significant drop.

If they could share more insights of IPs between the requests in the edge I could have tried to better map it to the nginx access.log on my side.

Patrick, can you email and I can maybe share more insights over email or chat? tobias.alriksson(@)gmail.com.

[pmeenan]

The big drop IS from a change. Previously 100% of the requests routed through workers showed as a cache hit. The change was SUPPOSED to show the actual cache status of requests that go through a worker.

The page rules are not an "alternative" solution, they go hand-in-hand and I strongly recommend doing it even without the worker. Both rules are needed (as part of a single page rule) to put Cloudflare's cache into "standards mode" where it operates like you'd expect a cache to operate.

Cache Everything: Name is a little confusing but this is really "evaluate all requests to see if they should be cached". Otherwise it only considers caching content based on the file extension (js, css, image extensions). I don't think this impacts the cache API the worker uses but it makes sense to always enable on every zone (only when the other flag is set).

Origin cache control: Says to "do whatever the server tells us to do in the cache-control and expires headers". Otherwise Cloudflare has default cache times that it uses depending on the content type.

The only reason you'd want to isolate the applications would be if one of them was causing the cache to purge on a regular basis. If they all use the same worker (and are all WordPress) then an update to any one of the sites will invalidate the cache for all 3 (which is fine if they are all mostly static). If you are using KV then it is possible to update the key name used to track the cache version to include the domain name in the key. That way all 3 sites would purge independently.

Feel free to email me (pmeenan @ webpagetest.org) but as much as possible 'd like to keep the discussion in an issue so it can be referenced by others later that may bump into the same issue.

[wpsumo]

@pmeenan

The page rules are not an "alternative" solution, they go hand-in-hand and I strongly recommend doing it even without the worker. Both rules are needed (as part of a single page rule) to put Cloudflare's cache into "standards mode" where it operates like you'd expect a cache to operate.

I'm aware of the rule but I did not see that this was supposed to be added int he setup installation of the workers. But you recommend using this one to make sure it is caching HTML? The worker will still do the job with bypass on cookies etc? The worries are that we, of course, do not want to cache when logged in so it causes issues and cache WP adminbar. Which is why I never would use this page rule if the worker wouldn't still make it bypass the cookies like in cloudflare business.

The wp admin bar is too handy to remove, but if it still should bypass as normal when adding this page rule surely I will do it! Just good to be on the safe side but did not know it go hand in hand wiht the worker and works as expected with bypass when the rule is enabled. I'm also wondering if flush of the HTML cache stored in the edge will be flushed and cleared correctly when using the page rule you suggest? Just want to be on the safe side as you can tell.

Cache Everything: Name is a little confusing but this is really "evaluate all requests to see if they should be cached". Otherwise, it only considers caching content based on the file extension (js, css, image extensions). I don't think this impacts the cache API the worker uses but it makes sense to always enable on every zone (only when the other flag is set).

I agree with the naming convention here. We want Cloudflare to work as cache should work as you say. Will set it up when we know the workers still make the HTML cache bypass on logged in and synchronize work with the rule, can you confirm? "(only when the other flag is set)." what do you mean?

Origin cache control: Says to "do whatever the server tells us to do in the cache-control and expires headers". Otherwise, Cloudflare has default cache times that it uses depending on the content type.

Ok, but why would this be needed if we already have set Browser Cache Expiration to respect the existing headers on the origin? But even here you recommend setting this page rule?

The only reason you'd want to isolate the applications would be if one of them was causing the cache to purge on a regular basis. If they all use the same worker (and are all WordPress) then an update to any one of the sites will invalidate the cache for all 3 (which is fine if they are all mostly static). If you are using KV then it is possible to update the key name used to track the cache version to include the domain name in the key. That way all 3 sites would purge independently.

Hmm update a post/page or publish a new trigger the flush of the cache right? Then it can become an unnecessary flush for many sites that are more static and are not having a frequent content flow.

We use KV, so you mean I would create unique Kv namespace? But don't they have the be named as they are to sync and work with the workers script? EDGE_CACHE = Wordpress Edge Cache

Or does it work to just change Wordpress Edge Cache to a unique per application? But the action in the workers EDGE_CACHE has to be kept to work with the script? Let me know an example. We, of course, want to keep the applications in one account but still would like their cache to be individual.

Feel free to email me (pmeenan @ webpagetest.org) but as much as possible 'd like to keep the discussion in an issue so it can be referenced by others later that may bump into the same issue.

Sure I just more want to share temporary access for inspecting and show website URLs which I prefer to send over private. But the issue I agree should be discussed here. So Conversation can still be in a github issue here but the confidential information should be kept private. Do you think this would be useful to share for further investigation?

[wpsumo]

@pmeenan Did you see my reply above?

[wpsumo]

@pmeenan Added the page rules to see if it change anything. I'm just worried this cause the frontend WP admin bar to be cached. But the worker should prevent this from happening even with the page rules?

Let's see in the next 24hours and more to see if it helps.

The only reason you'd want to isolate the applications would be if one of them was causing the cache to purge on a regular basis. If they all use the same worker (and are all WordPress) then an update to any one of the sites will invalidate the cache for all 3 (which is fine if they are all mostly static). If you are using KV then it is possible to update the key name used to track the cache version to include the domain name in the key. That way all 3 sites would purge independently

Would it be possible to set up and sort this, would be better even if the sites are more static than dynamic I want their edge cache to be treated individually.

The strange thing is sites using API is seeing 50% cache and sites using workers KV see below 20%

[wpsumo]

@pmeenan Last 7 days: Can not see any improvements after adding the suggested page rules for one of the applications in the shared workers KV.

None of the websites has updated plugins/themes/core or flushed cache manually.

I use Kinsta and just found out the cache lifetime by default is 1hour. This can be the trigger and root cause of the problem. The streamer and edge cache listen to the origin and then flush together with the nginx fastcgi cache module every our. Correct @pmeenan ?

Here a Kinsta, our servers use the nginx fastcgi cache module for page caching. And it is set to expire every 1 hour by default. However, clients can contact us if they need to increase or decrease this duration.

[wpsumo]

Latest update:

The issue persists, have the same issue with a ratio below 8%:

Cloudflares answer: < cf-cache-status: DYNAMIC

Dynamic means any requests we are not attempting to cache and are being sent to your origin server. We do not consider HTML can be cached by default. On the page my colleague just tested livecasinokings.com/, it is HTML content with a cookie in the response. To cache HTML with a cookie, you will need to use "Cache everything" along with "Edge Cache TTL".

The Workers script should solve the Edge Cache TTL. If we set it up as a page rule we face the issue with not listening to the cookies to bypass when logged in, and then cache the frontend WP admin bar.

@pmeenan @350d

[neostar]

@alriksson Not sure if it helps you, but I tried to add the Page Rule Cache Everything. This is what actually creates the trouble for me. The admin bar gets cached and I can even go to the backend in incognito mode. Very bizarre. I removed the page rule and purged the CF cache and W3TC cache. I'm now to back to just using the worker KV without CF page rules. Caching Level is set to Standard and Browser Cache Expiration in CF is set to 4 hours (Caching menu in CF).

I have the same CF-Cache-Status: Dynamic in Google Chrome but from the TTFB it's clear that the worker does its job and the request doesn't hit the origin server.

So in general I'm paying more attention the Chrome developer tools and what I see in the response header than what Cloudflare says in their statistics.

I'm using W3 Total Cache and the only thing I'm not sure of, is how to set the right Browser Cache parameters in W3TC. According to the Cloudflare Page Cache Plugin the CF worker stores the pages for one year, by modifying the caching parameters and on client request still replaces it with the original header. So, like you, this would invalidate my stored pages after 1 hour no matter how long they can be stored in the CF edge cache.

@pmeenan

[wpsumo]

@neostar My problem is solved and I think it was something on CF's side since no change and suddenly it just worked as it should. They never told me but must have been some update on their side.

I dont know but what works for me is having the cache everything page rule. Have no issues with WP admin bar being cached when using the worker and the cache everything helper plugin.

But you´re very much right with the analytics nto being 100% reliable but it is working for me now form 5% to 70-90% cache ratio. Have browser cache to listen to origin.

I do not use W3TC using Kinstas plugin and their origin cache. Modified the nginx page cache to store the html cache for a couple of hours on origin as well.

But as I said for me it just suddenly started to work on all websites after a couple of weeks of no changed on origin/application/cloudflare settings. Kinsta did not change anything according to their engineers. Can not tell what made it work, just happy it do work ;)

Just make sure to place page rules in the right order since it's a hierarchy:

[lauragift21]

Closing this issue because the initial issue was resolved. Also the templates have moved over to the worker-sdk monorepo: https://github.com/cloudflare/workers-sdk/tree/main/templates.

if you're still experiencing issues feel free to open an issue there.