web server keeps freezing up

williamcodes commented 8 years ago

Noelle got a 504 timeout error on Friday and I got one today. For me, it happened immediately after switching a hub to live mode.

cromulus commented 8 years ago

So, couple of things... Nginx is timing out serving static files:

Dec 11 23:01:16 setup.heatseeknyc.com docker[1060]: 208.125.11.210 - - [11/Dec/2015:23:01:16 +0000] "GET /static/favicon.ico HTTP/1.1" 504 584 "http://relay.heatseeknyc.com/hubs/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.80 Safari/537.36" "-"

So, perhaps moving static file serving to nginx might improve some of our timeouts: nginx.conf.sh:

cat << EOF
server {
  listen 80;
  # let nginx do what it does best: static files
  location / { try_files $uri @yourapplication; }

  location @yourapplication {
    include uwsgi_params;
    uwsgi_pass $APP_PORT_3030_TCP_ADDR:$APP_PORT_3030_TCP_PORT;
  }
}
EOF

Secondarily, the default setting for uwsgi_read_timeout is 60s.

What could possibly be taking 60 seconds?

My first thought is that perhaps something is restarting? the docker instances have been up for 20+ hours. Perhaps run another app instance?

Second thought: Flask blocking on IO. A long running request to heatseeknyc.com might block the thread and prevent a response? Perhaps putting gunicorn or somesuch in front of flask? Not sure if uwsgi-plugin-python3 is multithreaded or not, how it handles long running processes, etc.

williamcodes commented 8 years ago

This hasn't happened in a long time. Closing this issue for now.

heatseeknyc / relay

web server keeps freezing up #7