Reduce resource loads to help prevent site downtime and timeouts

rggibson commented 10 years ago

PB Tracker currently runs under Google App Engine's free tier usage, and every once in a while, traffic is high enough that we hit one or more of the free limits. Finding a way to reduce some of the resource usage would reduce the number of site outages. Particular resources that get used up often are data store reads and outgoing bandwidth. I think the best place to start would be to look at reducing the usage from /asup, although this will likely require some discussion with the TimeSplitTracker and LiveSplit devs. Here's what the site usage happens to look like at the time of posting this issue (forgive the horribly formatted print to pdf):

https://drive.google.com/file/d/0B5Hsa22FWRE6MmxRb2xMMEZpczg/edit?usp=sharing

trysdyn commented 10 years ago

That's interesting...

I ran Wireshark while doing a run tonight and took a look at the results after. The big culprit there seems to be that LiveSplit is very eager to update the gamelist from /asup, doing so every time I open the run editor and sometimes just when I expand the game list in LiveSplit itself. I myself yoinked about 1MB by doing one speedrun and editing my splits a bit after.

We could see if they want to add the If-Modified-Since HTTP header (if GAE supports it) and see if that'll cut it down a bit. That'd reduce most of those requests from 100k outgoing to a 2-3k.

Additionally, I notice that you're using HTML to resize boxarts (so they're anywhere from 10k to 300k each), and there doesn't seem to be If-Modified-Since support on the images either. So each time I view, for example. Duogongue's page (picking on him because he has the most games run on the site), I yank 5MB if I pan down the whole thing.

So yeah, my suggestion would be support If-Modified-Since and the 304 Not Modified response if at all possible.

If that's not possible, look into Cloudflare as a CDN. It'll do it for you and they have a free tier.

I'll take a look at GAE docs and see if it's possible.

trysdyn commented 10 years ago

I think I'm losing my mind. After making double and triple sure that I was getting a return code of "200 OK" on images, even if I had them cached, I went and read up on GAE docs to figure out how to increase cache lifetime. But now it seems I'm getting "304 Not Modified" returns from pbtracker.net.

So it's either intermittent that it's not properly delivering 304s, or I'm going crazy :)

rggibson commented 10 years ago

Cool, I didn't know about Wireshark. I have Google Analytics hooked up to the site, but I haven't checked on that in quite a while.

I knew LiveSplit was a big culprit and there is actually another issue related to this here: https://github.com/rggibson/pb-tracker/issues/47

Cloudflare is another service I hadn't heard of before. I've heard of Firebase and the two services might be similar, but I haven't tried anything like them myself. I like your suggestions and feel free to run with them. I wish I had more time myself to work on and improve PB Tracker, but it's just not in the cards for me right now.