Should images be served from CDN, Prod, or Static?

scd commented 11 years ago

[ ] Investigate costs - Adam thinks the following CDN is good value: http://www.incapsula.com/
[ ] How do we manage the propagation delay? Maybe serve from Prod for 1 hour?
[ ] Manage backups and unnecessary copies in dev?

brendanheywood commented 11 years ago

My over whelming vote is to either do nothing, or move photos to the static server. A CDN for this use case is an almost complete waste of time and money.

The 3 ways I am measuring each option are 1) how it will effect the render performance of a typical page, 2) how it will affect the number of concurrent users we can serve, and 3) effort / cost. For the typical best cases, photos are in the browser cache with correct expires so don't get served at all. All options below are for the worst case with a cold cache over a 3G network using stats from chrome dev tools.

Option 1: Do nothing

Render time: -
Concurrency -
Effort / cost: -

Option 2: Move photos and avatars to static server

Render time: The best improvement on the worst network is around 500ms. This will offload < 1% of www CPU load to the static server, so the main process could conceivably run max < 1% faster because it has less load, which means the main html page is served quicker. Even that is conservative because Apache uses almost no CPU to serve images, it is streamed directly off disk. It will have almost zero impact on render time, as most typical images are served in around 50ms, they don't block the page onload event, and in most cases aren't even visible 'above the fold' and are loaded before the user scroll's down. Bear in mind that no images are even fetched from the network or the cache until the main html page is served, which on a cold internal cache takes a couple seconds for a big node, or best case around 500ms to first byte, and 900ms for the complete html doc for a big node (eg Araps is around 1.2 - 2.0 seconds until the first byte). One small but real advantage of moving to the static server for photos is around 500 bytes of cookie payload one way x (4 photos + ~10 avatars) = 14 kb of data saved, which over a 3G network is around 500ms of download.
Concurrency: Negligible from our side for CPU, Memory. Possibly network bandwidth more if the static server is routed heaps to the www server but I don't know and doubt it.
Effort / cost: Simon's time

Option 3:

All advantages of a CDN are compared to option 2, not option 1.

Render time: A CDN could potentially have roughly 50ms reduced latency over broadband and maybe 200-500 reduce latency over 3G. Even in the worst case these are still going to be < 5% of the total page render.
Concurrency - It could handle way more photos, but because photo serving is not the bottleneck this is irrelevant. Our concurrency limits are throttled by the main html request.
Cost - $$ + time to config

Summary

Almost all the benefits of a CDN, in terms of compression, caching, gzipping, image header stripping, cookieless serving etc we already get from the static server. It is only the pipe size and the edge network benefits that's we're missing out on and neither of these are our bottle neck, not by a long way. All of the extra cool stuff like DDos attack protection I would be interested in, but we would only get these if we used a CDN in front of the main www server, as well as the static one. As a data point Incapsula's plans for this start at $300 a month. So essentially the cost is something, the benefit is essentially zero = a waste of money.

One pro a CDN gives us is SPDY support, but we can always turn that on for both the static server and www which is a very low hanging fruit see #667

Also worth noting that if we're talking a mobile device on a mobile network, the only images we're talking about are avatars, and the sponsor image. Everything else will get loaded on demand after the next release. So we are talking about a whole bunch of effort simply to offload worst case 10 avatars @ 2kb = 20kb + sponsor image = 12kb = grand total of 32kb of saving.

Also to answer your last two dot points, most CDN's don't have propagation delays, they are effectively proxies that sit in front of your origin server with dns changes. Most CDN architectures make each of their edge servers independent to make the system more robust, which means if a particular edge server has a cold cache for an item it goes back to your origin server which incurs the extra latency of that fetch (although many CDN's create long lived persistent connection to the origin to mitigate this (mega cool)). If you have lots of edge servers and very sparse content (eg long tail) you can get worse latency because the cache hits are too low. We've run into this on projects - you've gotta test it.

scd commented 11 years ago

It sounds like we get the benefit of your experience here.

My main driver is getting things off prod, but it looks like we may have solved the current prod bottle neck. There are possibly some other initiatives I will concentrate on first, so based on your comments I am happy to de-prioritise this for now.

Ultimately the saving in bandwidth is going to be significant for our end users so, we should cut it over to static eventually.

brendanheywood commented 11 years ago

I wouldn't say we've solved the prod bottle neck at all, just moved it to expose the next bottle neck. When I first started the bottle neck was absolutely front end rendering time but that's been essentially solved and so now the next bottle neck is the main www process. If and when we can get consistently back to say 100-200ms serving times instead of 2-3 seconds, then the next bottle neck will probably start to be network related. As we discussed I think facets can be sped up by at least an 1 or 2 orders of magnitude by moving to a star schema for reads. And there is probably some inefficiencies in list view from legacy data in the template which isn't being used.

brendanheywood commented 10 years ago

This is another low hanging fruit I think we can close off with little work. I was kinda horrified to discover that right now anything under /images/ isn't even served with an expires header including local avatars. Because they don't get an expires headers some browser are adding a guess expires, so an immediate refresh doesn't ask for them again but not great.

The obvious thing is to get these served from the static server. We could also improve the headers when served from prod too. I'm not whether all images can be treated as immutable, how do we handle image rotations when uploading? Do they get a new id or reuse the existing id/url?

scd commented 10 years ago

The ID and url stay the same. We could do a version number, but eventually we wanted to implement a hash. Potentially we also have this problem if we do any backend mods to the image.

Rotating only happens as part of the upload procedure so we could use an alternate URL which does not expire for that. This might be the easiest thing to do.

Are we even serving the images from static? I don't think I have completed this piece of work to server images from static.

This low hanging fruit may have a few extra tasks associated with it.

brendanheywood commented 10 years ago

No we are not serving from static, that's why I was asking about the rotation as I had a feeling that this wasn't sorted. I was thinking about hot fixing an interim short expires, like an hour or a day, but wasn't sure if this would mess up the upload + rotate process.

brendanheywood commented 10 years ago

See also #1603 - note fixing the expires header on topos and images served from www is the biggest performance win we can do right now

brendanheywood commented 10 years ago

theCrag / website