Caching, versioning, and CDNs

[svickers]

I've been using Combres for a few years and there is a gotcha when using a CDN that has always bugged me/caused headaches. In a scenario when your origin is pointing back to your production servers it is pretty easy get incorrect file version cached on the edge servers.

For instance if you have server pool A that has the following css, referenced as cdn.domain.com/combres/css/v1/

{body {color:#red}}

Now we want to change the text to blue so we update the file and deploy to server pool B cdn.domain.com/combres/css/v2/

{body {color:#blue}}

Now our QA team wants to verify the new text color on the servers before the load balancer is flipped. A tester pulls up one of the servers in his browser and sees all the body copy still in red, WTF! Not only do I have to deal with a new bug report but even worse we've just cached the old version of the file with the new file path. The CDN saw the request for /css/v2/, didn't have a copy of it, so proceeded to domain.com/combres/css/v2/ to download, which is the current red text version, damn.

This can be mitigated by not hitting any server before it is in production in this case, but it still happens sometimes. Also we have had problems in the past with both Azure and Appharbor not doing a 100% clean flip when new code is deployed, so traffic can still be served by out-of-date servers for some small period of time, and the likelihood of this happening tends to increase as the number of servers you have goes up. Flushing the CDN for all the files after each deployment is an option, but takes time, time that visitors can be seeing funky styles or perhaps completely broken javascript. Also during that window, the incorrect files will be cached on many browsers, a place where we have no control over forcing a flush, so they will have bad files for the foreseeable future or until we do a version bump and deploy, which starts the process over again.

Sorry for the long-winded example, fortunately there is an easy fix! Just don't send the cache headers if the version in the url does not match the current resource set version. See here: leafly-com/combres@62dfe7ac69

Let me know if you would like a pull request. Thanks for the continued work on this project.

[buunguyen]

Thanks for reporting the issue and the detailed write up. I will certainly be pleased to merge when we have worked out a good solution. I have two questions regarding the proposed fix:

1. It addresses client-side caching, but not serve-side caching? Shouldn't the same guard be placed in this line to prevent Combres to cache the response?
2. On the other hand, isn't a 404 response a better option than disabling caching and returning the wrong content? How does that impact your deployment workflow?

Let me know what you think.

[svickers]

1. Hmm, not sure. In our case we want it to cache because the initial combining takes a couple seconds and don't want the servers to be constantly churning on that. We will have multiple copies in the cache but the files will always be correct. If you are using a distributed cache I think this would cause a problem though, perhaps a switch in the config somewhere?
2. I would rather have the wrong styles than no styles, but maybe there is a case where that is preferred? Another configuration setting?

[buunguyen]

1. Distributed cache or not, wouldn't not guarding server cache mean the server will cache the response under the new path? That means even after you deploy the new files to the server in question, they won't be used until the cache expires, the same problem you tried to avoid, no? So I'm in the opinion that if a server hasn't been updated with new files, it shouldn't try to cache new path at all, both client & server.
2. I was thinking of scripts as well. If the HTML/DOM has been changed, using an old script might cause expected behavior. At least a 404 indicates that the server isn't ready with new code yet and nobody's surprised about some unexpected behavior (and filing a bug report).