search

yaoweibin / nginx_tcp_proxy_module

add the feature of tcp proxy with nginx, with health check and status monitor
http://yaoweibin.github.com/nginx_tcp_proxy_module
1.64k stars 365 forks source link

nginx freezing #60

Open alexbirkett opened 11 years ago

alexbirkett commented 11 years ago

Hi,

Since I've recompiled nginx (nginx-1.2.5) with the nginx_tcp_proxy_module (latest snapshot form github), the server has started freezing. It stopps accepting http requests. The web browsers are just left waiting. Restarting nginx fixes the problem. I'm running Ubuntu 12.04.1 LTS.

Here is the error log: 2012/12/11 11:35:26 [crit] 2657#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:26 [crit] 2658#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:26 [crit] 2658#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:28 [crit] 2657#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:28 [crit] 2657#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:28 [crit] 2658#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:28 [crit] 2658#0: accept4() failed (24: Too many open files) 2012/12/11 11:35:29 [error] 19566#0: 1 upstream servers are busy or encounter error! while connecting to upstream, client: 195.159.164.228, server: 0.0.0.0:80 2012/12/11 11:35:30 [error] 19566#0: 3 upstream servers are busy or encounter error! while connecting to upstream, client: 123.49.59.22, server: 0.0.0.0:80 2012/12/11 11:35:30 [error] 19566#0: *4 upstream servers are busy or encounter error! while connecting to upstream, client: 123.49.59.22, server: 0.0.0.0:80

The server receives a low volume of trafic. Do you know what is leaving file handles open and or why?

This may or may not be an issue with nginx_tcp_proxy_module.

Thanks,

Alex

yaoweibin commented 11 years ago

How many worker_connections do you set? What does this command show: ulimit -n ?

alexbirkett commented 11 years ago

ulimit -n returns 1024

nginx.conf contains:

events {
        worker_connections  1024;
}
yaoweibin commented 11 years ago

You should enlarge this two numbers. It's not suitable for production server. For example, you could specify them like this: ulimit -n 65536 worker_connections 10240

Maybe it's not enough to change these two value. You need change open file number the /etc/security/limits.conf for the owner of nginx worker.

alexbirkett commented 11 years ago

Yes I agree, I need to increase the maximum number of open file descriptors. My concern is that the descriptors are leaking, there are never 1024 simultaneously connected users. Could it be that if I increase the numbers, the same problem will occur but after a much greater period of time?

yaoweibin commented 11 years ago

On 2012-12-11 22:49, Alex Birkett wrote:

Yes I agree, I need to increase the maximum number of open file descriptors. My concern is that the descriptors are leaking, there are never 1024 simultaneously connected users. Could it be that if I increase the numbers, the same problem will occur but after a much greater period of time?

I can't guarantee it. If that happens, you could report me again.

Note that you use tcp proxy which will open two connections for one user. One for the frontend client, one for backend server.

Thanks.

Thanks, -Weibin Yao

alexbirkett commented 11 years ago

Ok. Let's do that. Thanks very much for your help. Please close the issue.

yaoweibin commented 11 years ago

You can close it yourself. let's wait for a while, hope it can solve your problem.