http://browsershots.org/screenshots/*/ is far too slow (lots of Gateway Timeout)

[GoogleCodeExporter]

Calling e.g.
http://browsershots.org/screenshots/24e999bab51e658deb86ae903505cae4/ is
very slow and when having multiple tabs opening such URLs, there is a lot
of "504 Gateway Timeout" responses.

To fix this, I propose:
1. Improve the code in question (is it available somewhere?)
2. Update the software involved (it uses e.g. nginx/0.5.33 after all)

I've also noticed that there are far too many "The browser has not visited
the page" errors, probably caused by api.browsershots.org not responding in
a timely manner to the request to redirect to the target site.
This is an urgent issue and probably related/similar to the initial report.

If you need more hardware/processing power, please let me know - I can help
you out probably, although it looks like software fixes could make it a lot
better probably already.

Original issue reported on code.google.com by dhahler@gmail.com on 2 Feb 2010 at 8:21

[GoogleCodeExporter]

Right now during peak hour the site has 150+ requests per hour.
With 50 browsers, that's about 2+ uploads per second.  If you count the polling 
for the url, authentication, upload, that's several hundred requests every time 
someone presses the submit button on the front page.

I seem to be seeing a lot of "503 overloaded" errors during the peak hours.

The current rule is to select the latest minor version from each major version 
of browsers.  ie. 3.6 but not 3.5, 3.0.  This is not accurate since no one uses 
firefox 1, 2 but a lot use 3.0, 3.5.
If no browsers were ticked on the front page by default then there'll be less 
load on the web server.  Or maybe a random selection, since poor MSIE will get 
overloaded.

Original comment by nik...@gmail.com on 20 Jan 2011 at 3:38