search

unbit / uwsgi

uWSGI application server container
http://projects.unbit.it/uwsgi
Other
3.45k stars 691 forks source link

Connection timed out and Resource temporarily unavailable #1647

Open haytham-salhi opened 7 years ago

haytham-salhi commented 7 years ago

Hello folks!

I have a python web app running on EC2 instance (Ubuntu OS). I have nginx as proxy server communicating with the uwsgi via Unix sockets. Even though the number of incoming requests are not that high, sometimes the communication between nginx and uwsgi gets lost with the following logs from nignx error.log (See below) and then after some period it works normally without restarting anything!

Here's my nginx conf file: my-nginx.conf.txt

Here's my uwsgi ini file: uwsgi.ini.txt

uwsgi version: Latest: 2.0.15

2017/09/22 18:02:02 [error] 6536#0: *56531 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.10.2.171, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock", host: "10.10.2.171"
2017/09/22 18:03:02 [error] 6536#0: *56603 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.10.2.171, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock", host: "10.10.2.171"
2017/09/22 18:04:01 [error] 6536#0: *56691 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.10.2.171, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock", host: "10.10.2.171"
2017/09/22 18:04:52 [error] 6536#0: *56841 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.22, server: communication-api.harri.com, request: "GET /api/v1/brands/68809/notifications/counts HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "communication-api.harri.com", referrer: "https://harri.com/team"
2017/09/22 18:04:59 [error] 6536#0: *56816 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.153, server: communication-api.harri.com, request: "GET /api/v1/brands/34821/notifications/counts?brand_id=34821 HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "communication-api.team.live"
2017/09/22 18:05:01 [error] 6536#0: *56777 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.10.2.171, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock", host: "10.10.2.171"
2017/09/22 18:05:01 [error] 6536#0: *56858 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.2.171, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:03 [error] 6536#0: *56860 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.5.151, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:03 [error] 6536#0: *56862 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.22, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:03 [error] 6536#0: *56864 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.4.254, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:03 [error] 6536#0: *56866 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.153, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:03 [error] 6536#0: *56868 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.5.153, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:03 [error] 6536#0: *56870 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.4.70, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:08 [error] 6536#0: *56816 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.153, server: communication-api.harri.com, request: "GET /api/v1/brands/69204/posts?fetch=after&limit=10 HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "communication-api.team.live", referrer: "https://team.live/availability"
2017/09/22 18:05:13 [error] 6536#0: *56880 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.22, server: communication-api.harri.com, request: "GET /api/v1/brands/949084/posts?fetch=after&limit=10 HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "communication-api.harri.com", referrer: "https://harri.com/schedule?date=2017-09-25"
2017/09/22 18:05:23 [error] 6536#0: *56880 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.22, server: communication-api.harri.com, request: "GET /api/v1/brands/68809/notifications/counts HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "communication-api.harri.com", referrer: "https://harri.com/team"
2017/09/22 18:05:27 [error] 6536#0: *56880 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.22, server: communication-api.harri.com, request: "GET /api/v1/brands/829255/conversations/2562/messages?limit=20 HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "communication-api.harri.com", referrer: "https://harri.com/schedule?date=2017-09-25"
2017/09/22 18:05:29 [error] 6536#0: *56880 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.22, server: communication-api.harri.com, request: "GET /api/v1/brands/883214/conversations HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "communication-api.harri.com", referrer: "https://harri.com/team"
2017/09/22 18:05:33 [error] 6536#0: *56860 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.5.151, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:33 [error] 6536#0: *56862 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.22, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:33 [error] 6536#0: *56864 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.4.254, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:33 [error] 6536#0: *56866 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.153, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:33 [error] 6536#0: *56868 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.5.153, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:05:33 [error] 6536#0: *56870 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.4.70, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:03 [error] 6536#0: *56864 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.4.254, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:03 [error] 6536#0: *56866 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.153, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:03 [error] 6536#0: *56868 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.5.153, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:03 [error] 6536#0: *56870 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.4.70, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:33 [error] 6536#0: *56908 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.5.151, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:33 [error] 6536#0: *56910 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.22, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:33 [error] 6536#0: *56864 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.4.254, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:33 [error] 6536#0: *56866 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.6.153, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:33 [error] 6536#0: *56868 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.5.153, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
2017/09/22 18:06:33 [error] 6536#0: *56870 connect() to unix:///var/sockets/harri.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 10.10.4.70, server: communication-api.harri.com, request: "GET /health_check HTTP/1.1", upstream: "uwsgi://unix:///var/sockets/harri.sock:", host: "10.10.2.171"
pilgrim2go commented 6 years ago

@haytham-salhi : how is your issue going now?

I had similar logs when i have default uwsgi connections is 100( Queue size for uWSGI is by default 100) and I started 200 requests simultaneously. By changing [uwsgi] listen=400 solved the issue ( or started less than 100 requests my case)

haytham-salhi commented 6 years ago

Thanks for asking, @pilgrim2go!

Although the queue size could be a reason, the issue was not related to the number of queued connections. The workers of uwsgi were no longer responding to new requests because they had already been suspended with "in-processing" requests. Like a deadlock, you can say! I resolved the issue by running the uwsgi in harakiri mode whereby a request timeout is set so that the worker does not get suspended in case the request takes too long!

jin09 commented 6 years ago

Default queue size of uwsgi is 100 and max allowed by kernel is 128. If you increase the default queue size of uwsgi make sure you increase the kernel's limit as well. You can do that by changing the value of /proc/sys/net/core/somaxconn (for linux based operating systems)