Open kcd83 opened 9 years ago
Do you see this issue only with managed listener? Did you try NativeListener?
With managed listener: what if you change keepalive
settings in nginx to 60? What is the state of sockets when number of sockets are close to maximal allowed?
For the NativeListener I need libevent2 which I am avoiding due to a conflict we had previously. I could revisit this.
All connections claim to be connected and I got up to 64k of them. When generating traffic I do see some connections come and go but most stay open forever.
Yes I can try 60s keepalive On 17/03/2015 8:32 PM, "Sergey Zhukov" notifications@github.com wrote:
Do you see this issue only with managed listener? Did you try NativeListener?
With managed listener: what if you change keepalive settings in nginx to 60? What is the state of sockets when number of sockets are close to maximal allowed?
— Reply to this email directly or view it on GitHub https://github.com/xplicit/HyperFastCgi/issues/23#issuecomment-82182847.
With keepalive 60 and ab concurrent connections 50 nginx should not create any additional connections so the number of opened connection should not grow. If it grows it is probably something wrong with nginx installation. If it does not grow then the issue on HFC/mono side. I did not see such issue on Ubuntu system, but I'll recheck it.
Ah I see what is happening now thanks. With that scenario you are correct I have no problem.
However I read up on this:
It should be particularly noted that the keepalive directive does not limit the total number of connections to upstream servers that an nginx worker process can open. The connections parameter should be set to a number small enough to let upstream servers process new incoming connections as well.
Obviously my test was an extreme example but the occasional extra connection nginx opens is never closed (fcgi_keep_conn on
). In a long running system under load will eventually suffer the same problem. Also if nginx is restarted often. If I can find time tomorrow I will try some more things out.
A quick update. I toyed with some changes (hacks) to clean up stale connections with some success but moved on with performance testing using the higher number of keepalive sockets workaround above.
Using a small vagrant vm and apache bench i get:
helloworld equivalent - something static
in-memory-response - something cpu bound with no other i/o like database
So perhaps the better solution (when nginx and hyperfastcgi are co-located) is to turn off keepalives.
As for this bug report, I feel like this is an nginx issue where by nginx should be configurable to not send fastcgi_keep_conn on
for sockets greater than keepalive
. If it wasn't for the above results I was hoping to try some other fastcgi implementation as a comparison.
On a wild tangent, be very wary of the default MONO_EVENTLOG_TYPE="local:/var/log/somewhere"
... in one scenario i was logging process times and the above dropped to just 35 r/s ...!
Hello. Great work with the project. Unfortunately I am having issues with the ManagedFastCgiListener under CentOS 6.
Have you seen this problem where many sockets are not closing?
I can reproduce with
ab -t60 -n 50000 -c 50 'http://somehost'
and watching the sockets/file descriptors climb (netstat -ap | grep mono | wc -l
andls -l /proc/
pidof mono/fd
) until hyperfastcgi hits the ulimit and crashes.hyperfastcgi
nginx
mono is a custom build of 3.2 which is not the newest so I haven't ruled that out yet.