getgrav / grav

Modern, Crazy Fast, Ridiculously Easy and Amazingly Powerful Flat-File CMS powered by PHP, Markdown, Twig, and Symfony
https://getgrav.org
MIT License
14.59k stars 1.41k forks source link

Default browser cache settings conflict with Grav cache for assets generated by assets pipelines #2992

Open nbusseneau opened 4 years ago

nbusseneau commented 4 years ago

Hello,

Description

I have encountered an interesting cache issue with Grav v1.6.26 using default Grav settings, with the exception of enabling assets pipelines through system.yaml:

assets:
  css_pipeline: true
  js_pipeline: true

I noticed that on the first visit of the day, CSS and JS assets would not load properly, resulting in a broken website. Inspecting network requests revealed that several GET requests for assets were failing with 404. Upon page reload the same requests succeed with 200 and next reloads yield 304, thus re-using browser cache.

Investigation

I thought this might be due to the default cache clear job scheduled every night, and confirmed the hypothesis by reproducing the issue reliably with the following protocol (prerequisite: system.yaml above):

At first I thought this was because CSS/JS pipelines were too slow, and for some reason Grav was rendering the page and returning it the client with CSS/JS links to assets before these same assets were instantiated.

To test this, I adapted the test protocol to work with multiple clients:

As we can see, this hypothesis was incorrect as client 2 failed just like client 1 did, even though assets were properly generated following client 1 first access.

I then realized that CSS/JS filenames generated from pipelines were stable, i.e. not changing upon cache clear. A page that has not changed in content will always have the same CSS/JS assets filenames. So I thought this might be due to an unfortunate interaction between browser cache and Grav assets cache.

To test this, I modified my system.yaml to reduce expiration time significantly:

assets:
  css_pipeline: true
  js_pipeline: true

pages:
  expires: 10

And then adapted the test protocol:

Thus, it seems to confirm that the default Expires header of 7 days conflict with the daily Grav cache clear when using assets pipelines.

What I find very strange is that I could not reproduce the issue when assets pipelines were disabled: instead of 404, server correctly returns 304 and thus browser cache kicks in. This might indicate a server-side configuration issue. I will investigate further when I have some time.

Mitigation

Obvious workarounds are easy to implement. Take your pick:

tl;dr

Browser cache has some sort of conflict with Grav cache for assets generated by assets pipelines. This breaks pages every time a Grav cache clear occurs before browser cache has expired, which with default settings is once a day (Grav cache cleared every night, whereas browser cache expiration defaults to 7 days).

What is strange is that it does not seem to happen when assets pipelines are disabled, even though logically there should not be any difference.

rhukster commented 4 years ago

My hunch from your initial description of your issue was that you are probably including a bunch of external assets in your pipeline, and the process of retrieving/compressing/minifying these is timing out or just failing. It seems likely that it is a network issue if it doesn't work 'sometimes', but then later works. Would really need to get a better idea of your asset setup. perhaps even including a copy/paste of your <head> tag with pipeline off so we can see all the assets listed? Better would be to see the Twig where you load the assets but that might not be possible as it could be spread out over multiple files.

I would suggest if you have external assets to remove those from the pipeline and try again.

nbusseneau commented 4 years ago

Here is a copy of all CSS and JS assets in my <head> tag:

<link href="/user/plugins/markdown-notices/assets/notices.css" type="text/css" rel="stylesheet">
<link href="/user/plugins/form/assets/form-styles.css" type="text/css" rel="stylesheet">
<link href="/user/plugins/langswitcher/css/langswitcher.css" type="text/css" rel="stylesheet">
<link href="/user/plugins/mathjax/assets/css/mathjax.css" type="text/css" rel="stylesheet">
<link href="/user/plugins/simplesearch/css/simplesearch.css" type="text/css" rel="stylesheet">
<link href="/user/plugins/prism-highlight/css/prism.css" type="text/css" rel="stylesheet">
<link href="/user/plugins/prism-highlight/css/themes/prism-base16-bright.dark.css" type="text/css" rel="stylesheet">
<link href="/user/plugins/featherlight/css/featherlight.min.css" type="text/css" rel="stylesheet">
<link href="/user/plugins/login/css/login.css" type="text/css" rel="stylesheet">
<link href="/user/themes/custom-quark/css-compiled/spectre.min.css" type="text/css" rel="stylesheet">
<link href="/user/themes/custom-quark/css-compiled/spectre-exp.min.css" type="text/css" rel="stylesheet">
<link href="/user/themes/quark/css-compiled/spectre-icons.min.css" type="text/css" rel="stylesheet">
<link href="/user/themes/custom-quark/css-compiled/theme.min.css" type="text/css" rel="stylesheet">
<link href="/user/themes/custom-quark/css/custom.css" type="text/css" rel="stylesheet">
<link href="/user/themes/custom-quark/css/line-awesome.min.css" type="text/css" rel="stylesheet">

<script src="/system/assets/jquery/jquery-2.x.min.js"></script>
<script src="/user/plugins/mathjax/assets/js/mathjax.js"></script>
<script src="/user/plugins/featherlight/js/featherlight.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script>
$(document).ready(function(){
    $('a[rel="lightbox"]').featherlight({
        openSpeed: 250,
        closeSpeed: 250,
        closeOnClick: 'background',
        closeOnEsc: '1',
        root: 'body'
    });
});
</script>

I use the Quark theme, so all of these are loaded via base.html.twig through the Asset Manager.

When enabling pipelines, here is what <head> looks like:

<link href="/assets/20d564efdecc7734496fbe04722fc399.css" type="text/css" rel="stylesheet">

<script src="/assets/8a5e312f49c642b5c40ba888e2904aaa.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script>
$(document).ready(function(){
    $('a[rel="lightbox"]').featherlight({
        openSpeed: 250,
        closeSpeed: 250,
        closeOnClick: 'background',
        closeOnEsc: '1',
        root: 'body'
    });
});
</script>

You'll note that I had already disabled externals inclusion (the third-party MathJax plugin does not work when its JS is merged, this is something I have noted for a later investigation as I'm unsure if it is a bug on Grav's end or the MathJax plugin's end).

Thus, I don't think this is a network-related issue, since there is no external exclusion at all.

One thing to note:

It seems likely that it is a network issue if it doesn't work 'sometimes', but then later works.

It is not "sometimes", I can reliably make it "not work" with the protocol above.

Are there any logs I could check, or specific parts of the code where I could insert logging / debugging dumps, to help troubleshoot the generation process?

nbusseneau commented 4 years ago

Oh, another thing that reinforces my hunch that it is a conflict with browser cache: as stated in OP, we can clearly see in the network frames that the site breaks because the older js and css fail with 404. My hunch was that this happens because the "cached version" does not exist anymore on the server's end.

When setting enable_asset_timestamp to true, this is even easier to spot: image

rhukster commented 4 years ago

Well, i am really am not sure how to recreate your issue. I'm using latest Grav 1.7, and on my rather large test 'sandbox' site which has over 50 plugins, many of them have their own CSS and JS. I have pipeline enabled for both JS and CSS and even after cache clear, the pipelined files are regenerated just fine. no 404s. Firefox 80.0.1 on mac btw:

Home  Grav 2020-09-17 at 2 23 14 PM
nbusseneau commented 4 years ago

Perhaps it originates from the HTTP server? I use nginx.

Do you have a snapshot I could use to setup an identical test server, and see if I reproduce the issue with it? Otherwise I can try to make one out of a clean Grav install.

julienloizelet commented 2 years ago

Seems that I have the same issue. But I guess I can't help much ... Only thing I can say is:

How I reproduce the bug each time :

On Firefox console, I can read something like:

Failed to load for <script> element whose source is "https://mydomain.com/assets/4a0e19561080fbbee6bb0b44d326c840.js

Bug disappears only if I disable JS pipeline.

nbusseneau commented 1 year ago

No I don't think anybody ever got bothered enough to look into it further. I've been running with assets pipelines disabled forever now, it doesn't bother me :D