Occasionally our assets fail to load and a return a 404 (observed on https://archive.matrix.org/). It seems like this spawns from our origin server serving a 404 : Cannot GET /assets/styles-63d89b7d.css (Express error). So for some reason the express.static(...) call can't find our assets on disk.
This is a bit confusing though because Cloudflare should be caching our assets forever (for 1-year) after seeing a successful response. I even see the Cache-Control: max-age=31536000 header in place and responds with Cf-Cache-Status: HIT indicating that Cloudflare is using the cache.
Potential solutions
Find the root cause
What's causing Express to not be able to find the assets on disk sometimes?
Always serve the /assets from Nginx
We could maybe workaround the problem by serving the /assets directory directly from nginx instead of relying on Express. I'd rather find the root cause first in any case.
Why isn't Cloudflare serving the cached response?
To be determined
Or what's causing Cloudflare to bust it's own cache when our origin server starts spitting out a 404?
Occasionally our assets fail to load and a return a 404 (observed on https://archive.matrix.org/). It seems like this spawns from our origin server serving a
404
:Cannot GET /assets/styles-63d89b7d.css
(Express error). So for some reason theexpress.static(...)
call can't find our assets on disk.https://github.com/matrix-org/matrix-public-archive/blob/5caf9dc1b8d5e98fdc99ab06ab9b9d09d2f546ed/server/routes/install-routes.js#L42-L43
Cloudflare will cache our 404 responses for 3 minutes. And things seem to come back after the cache expires.
This is a bit confusing though because Cloudflare should be caching our assets forever (for 1-year) after seeing a successful response. I even see the
Cache-Control: max-age=31536000
header in place and responds withCf-Cache-Status: HIT
indicating that Cloudflare is using the cache.Potential solutions
Find the root cause
What's causing Express to not be able to find the assets on disk sometimes?
Always serve the
/assets
from NginxWe could maybe workaround the problem by serving the
/assets
directory directly from nginx instead of relying on Express. I'd rather find the root cause first in any case.Why isn't Cloudflare serving the cached response?
To be determined
Or what's causing Cloudflare to bust it's own cache when our origin server starts spitting out a 404?