search

jywarren / plots

This is the old website for Public Lab; visit http://publiclab.org and https://github.com/publiclab/plots2 for the new website.
http://old.publiclab.org
12 stars 0 forks source link

optimize serving balloon-mapping and /home pages for high traffic #147

Open jywarren opened 12 years ago

jywarren commented 12 years ago

Currently researching how to use NGINX: http://www.howtoforge.com/how-to-speed-up-drupal-7.7-with-boost-and-nginx-debian-squeeze-p3

To get actual IP addresses instead of the reverse proxy, we'll need to install rpaf: http://olex.openlogic.com/wazi/2011/add-a-nginx-reverse-proxy-to-your-lamp-setup/

jywarren commented 12 years ago

Also remove www.pub... and http://publiclab redundancy (see howtoforge article above, p2)

rjcorwin commented 12 years ago

That install doesn't look too hairy. I'm interested to see how it goes. Are you working on it right now?

RJ Steinert http://rjsteinert.com

On Wed, Apr 18, 2012 at 4:56 PM, Jeffrey Warren < reply@reply.github.com

wrote:

Currently researching how to use NGINX: http://www.howtoforge.com/how-to-speed-up-drupal-7.7-with-boost-and-nginx-debian-squeeze-p3

To get actual IP addresses instead of the reverse proxy, we'll need to install rpaf: http://olex.openlogic.com/wazi/2011/add-a-nginx-reverse-proxy-to-your-lamp-setup/

Reply to this email directly or view it on GitHub: https://github.com/jywarren/plots/issues/147

jywarren commented 12 years ago

yep; i'm in the public chat room if you're around: publiclaboratory.org/chat

On Wed, Apr 18, 2012 at 5:11 PM, R.J. Steinert < reply@reply.github.com

wrote:

That install doesn't look too hairy. I'm interested to see how it goes. Are you working on it right now?

RJ Steinert http://rjsteinert.com

On Wed, Apr 18, 2012 at 4:56 PM, Jeffrey Warren < reply@reply.github.com

wrote:

Currently researching how to use NGINX:

http://www.howtoforge.com/how-to-speed-up-drupal-7.7-with-boost-and-nginx-debian-squeeze-p3

To get actual IP addresses instead of the reverse proxy, we'll need to install rpaf:

http://olex.openlogic.com/wazi/2011/add-a-nginx-reverse-proxy-to-your-lamp-setup/

Reply to this email directly or view it on GitHub: https://github.com/jywarren/plots/issues/147

Reply to this email directly or view it on GitHub: https://github.com/jywarren/plots/issues/147#issuecomment-5208910

jywarren commented 12 years ago

ok, i'm currently blocking hotlinked files from off domain, 'cause that's in the example nginx config. Reminder to self to re-enable that.

jywarren commented 12 years ago

ok running nginx at publiclaboratory.org:8088 but got:

2012/04/18 17:51:33 [error] 5660#0: *1 no live upstreams while connecting to upstream, client: 208.54.36.199, server: publiclaboratory.org, request: "GET /index.php HTTP/1.1", upstream: "http://localhost/index.php", host: "publiclaboratory.org:8088"
2012/04/18 17:52:38 [error] 5660#0: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 208.54.36.199, server: publiclaboratory.org, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:80/", host: "publiclaboratory.org:8088"

maybe i have localhost access off for apache? working...

jywarren commented 12 years ago

i got nginx working and redirecting, but when i try to swap it so nginx is handling port 80 and apache on port 8088, it says port already in use... i think this means i'd have to swap over ALL apache sites to 8088, individually in the /etc/apache2/sites-available/* directory to all say 8088, and nginx would sit in front of them.

or I could try to make the /etc/apache2/sites-available/default file deal with everything except publiclaboratory.org:80

either way it'll take a little work to get going and I'm not really wanting to try it on the production server. Load times have dropped again as traffic has gone down so I want to test this out on a smaller service like SpectralWorkbench.org before really trying it on publiclaboratory.org.

jywarren commented 12 years ago

alternative: proxy everything through NGINX and run apache on port 8088

this means figuring out how to proxy each site to its respective hostname in /etc/nginx/sites-available/*?

multiple server entries seems dumb, can we wildcard? (http://www.besttechie.net/wp-content/uploads/nginx-example.conf)

jywarren commented 12 years ago

also someone says (http://wp-performance.com/2010/10/nginx-reverse-proxy-cache-wordpress-apache/):

#Probably not needed, as the proxy will pass back the host in "proxy_set_header"
server_name www.yoursite.com yoursite.com;
access_log /var/log/nginx/yoursite.proxied.log;

or wildcard:

http://nginx.org/en/docs/http/server_names.html

server {
    listen       80;
    server_name  *.*;
    ...
}
rjcorwin commented 12 years ago

Interesting stuff.

proxy everything through NGINX and run apache on port 8088

I wonder if this would end up being difficult having to tune /etc/nginx/sites-available/default for every site on the server...

if ($http_cookie ~ "DRUPAL_UID" )

Would have to be expanded. I wonder what else...

RJ Steinert http://rjsteinert.com

On Thu, Apr 19, 2012 at 9:23 AM, Jeffrey Warren < reply@reply.github.com

wrote:

also someone says ( http://wp-performance.com/2010/10/nginx-reverse-proxy-cache-wordpress-apache/ ):

Probably not needed, as the proxy will pass back the host in

"proxy_set_header" server_name www.yoursite.com yoursite.com; access_log /var/log/nginx/yoursite.proxied.log;

Reply to this email directly or view it on GitHub: https://github.com/jywarren/plots/issues/147#issuecomment-5221367

jywarren commented 12 years ago

i'm trying it now; still can't seem to allow connections to Apache via 127.0.0.1 or localhost... curl blablabla 127.0.0.1:80 fails

also:

2012/04/19 16:41:08 [error] 30686#0: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 18.248.6.170, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:80/", host: "spectralworkbench.org:8088"
2012/04/19 16:41:09 [error] 30686#0: *9 open() "/opt/nginx/html/favicon.ico" failed (2: No such file or directory), client: 18.248.6.170, server: , request: "GET /favicon.ico HTTP/1.1", host: "spectralworkbench.org:8088"

when trying to get to spectralworkbench.org:8088

jywarren commented 12 years ago

almost there, getting to apache but need to get the virtual host config to actually direct it to the right web app. This has a lot of helpful stuff: http://www.dikant.de/2008/07/10/nginx-as-a-reverse-proxy-for-apache/

jywarren commented 12 years ago

finally, reverse proxying directly to rails apps: http://blog.phusion.nl/2010/09/21/phusion-passenger-running-multiple-ruby-versions/

jywarren commented 12 years ago

close to ready to go, but I want to do it from a hardline (i'm on a train) like in the Matrix. The nginx default entries look like this:

server {
        listen 80;
        server_name archive.publiclaboratory.org;
        location / {
                proxy_pass http://archive.publiclaboratory.org:8088;
                proxy_set_header Host $host;
        }
}

server {
        listen 80;
        server_name mapknitter.org;
        location / {
                proxy_pass http://mapknitter.org:8088;
                proxy_set_header Host $host;
        }
}
  1. backup whole /etc/apache2/sites-available directory and /etc/nginx/sites-available/default to .bkp
  2. change all instances of 8088 > 80 and 80 > 8088 in /etc/nginx/sites-available/default
  3. change all instances of 80 > 8088 and 8088 > 80 in /etc/apache2/sites-available/*
  4. take down apache & nginx, restart both
  5. check each domain

I'd like to benchmark before and after, but that might be hard on a production machine. If we don't have more crazy server issues (we're on slashdot front page now and doing OK, i guess!) I can let the current system run for a couple days to let it even out, then do this swich. I'd hope to see a distinct drop in response time. In very rough tests, I've seen Nginx serve an HTML page between 60-100% faster than Apache. Nice.