Limited to 1024 concurrent connections, and looking for suggestions.

[rmmoul]

I started a project using ratchet, and wanted to test the number of connections that could be handled at one time on our server (Digital Ocean Ubuntu 14.04, 2 cores, 4GB ram running php 5.6.7 and apache2 2.4.7).

I followed some of the suggestions here on the deploy page http://socketo.me/docs/deploy to help increase the number of connections that could be handled, and seemed to get the ulimit and such to up the number of open files to 10,000.

I started running tests today using thor (https://github.com/observing/thor):

thor --amount 10000 ws://example.com:2600 -C 1000 -W 2 -M 100 

I got a php error when the number of connections exceeded 1024:

PHP Warning:  stream_select(): You MUST recompile PHP with a larger value of FD_SETSIZE.
It is set to 1024, but you have descriptors numbered at least as high as 1123.
 --enable-fd-setsize=2048 is recommended, but you may want to set it
to equal the maximum number of open files supported by your system,
in order to avoid seeing this error again at a later date. in
/var/www/example.com/server/vendor/react/event-loop/StreamSelectLoop.php on line 255

I was actually using php 5.5.9 at the time, so I followed some old instructions from http://ubuntuforums.org/archive/index.php/t-2130554.html and increased the FD_SETSIZE value to 10000 in the following two files and then downloaded and compiled php 5.6.7.

/usr/include/linux/posix_types.h
/usr/include/x86_64-linux-gnu/bits/typesizes.h

That coupled with using this command to run the server through supervisor:

bash -c "ulimit -n 10000 && php /var/www/hyvly.com/server/server.php"

Seems to have allowed the number of connections to go beyond 1024, but now it causes a buffer overflow within php, showing this error in the log file before restarting the process:

*** buffer overflow detected ***: php terminated

I'm curious how other users are getting beyond 1024 concurrent connections, whether some of you have never hit this limit at all (could you share your environment details), or made certain changes to get beyond it (could you share what changes you've made)?

[kinncj]

Does this server only handles the socket application?

1024 seems to be a really low number of users.

My guess is: your code may be doing something really weird.

Try to profile your application in order to see where it's overwriting the adjacent memory. If it's the vendor (ratchetphp), report as a bug with more details (such as what are you really doing with it...) ; otherwise, fix the gap on your app.

Worst case scenario: Bring more nodes to your application and use a shared memory (a.k.a memcached, riak, etc etc etc) to share the state between the nodes in order to scale (very last option; IMHO 1024 users are nothing... unless you keep their messages in memory and this messages are significant size BLOBs).

[rmmoul]

Yeah, I thought that seemed low too.

The application is a simple chat application, and it's all that's running on the server. When the concurrent connections are under 1024 (where php craps out) the server doesn't get strained.

[kinncj]

Are you storing the messages in memory somehow?

May be related.

[rmmoul]

Not for this round of tests, just incrementing and decrementing a connection counter.

[kinncj]

Can you profile (xhprof, whatever) and share the reports? I'm interesting to see the results!

[lokielse]

Limited to about 50 concurrent connections, and looking for suggestions.

Run command:

echo "show info" | socat /tmp/haproxy.sock stdio

Result: Name: HAProxy Version: 1.5.3 Release_date: 2014/07/25 Nbproc: 1 Process_num: 1 Pid: 21515 Uptime: 0d 0h13m30s Uptime_sec: 810 Memmax_MB: 16 #Here Memmax_MB is 16M, I don't know how to increase it. Ulimit-n: 20032 Maxsock: 20032 Maxconn: 10000 Hard_maxconn: 10000 CurrConns: 53 #Current connections count CumConns: 278 CumReq: 278 Maxpipes: 0 PipesUsed: 0 PipesFree: 0 ConnRate: 2 ConnRateLimit: 0 MaxConnRate: 14 SessRate: 2 SessRateLimit: 0 MaxSessRate: 14 CompressBpsIn: 0 CompressBpsOut: 0 CompressBpsRateLim: 0 Tasks: 52 Run_queue: 1 Idle_pct: 100

[lokielse]

I found that my init.d/haproxy start haproxy with start-stop-daemon which result of 16M memory limit

I change it to this and work

haproxy_start()
{
    $HAPROXY -f "$CONFIG" -D -p "$PIDFILE"
    return 0
}

[hsvikum]

@rmmoul I'm having theexact same issue my chat server fails at 1019 connections.I increased the allowed file opens.and compiled php with the necessary configurations.but it seems that php is still not detecting the changed amount.any luck fixings this.

[benconnito]

@rmmoul I ran into the 1024 limit and did NOT want to compile php and run my own version. So to get around the 1024 limit, i ran multiple instances on several ports and had a http endpoint that round robined the ports to the client.

[rmmoul]

@benconnito That's what I've been considering doing as well. I was hoping to keep them all on the same port, though, to avoid needing to set up a redis pub/sub server to keep all of the clients connected. I think you've found the easiest / best solution.

@hsvikum I haven't found a way to get around this through recompiling or changing up my server's config options. @benconnito's solution is probably the way to go, though I haven't tried what @lokielse suggested to up the memory limit (his connection limit was super small though).

[lokielse]

Try to Increase Open Files Limit

https://rtcamp.com/tutorials/linux/increase-open-files-limit/

[Mecanik]

I did everything mentioned here on Linux, and my application is limited to 254 connections no matter what. I have optimised PHP and Apache, but still no luck. I also tested on my Windows dev enviroment, same result. My application is a multi-room chat server... very simple. Any suggestions ?

[WyriHaximus]

@Mecanik use a event loop other then the default, which is limited to 1024 open file desciptors: https://github.com/reactphp/event-loop#loop-implementations

[Mecanik]

Well I do not quite understand how that works ... but I will try. This is how I have it now:

$wsServer   = new WsServer($chatServer);
        $wsServer->disableVersion(0); // old, bad, protocol version

        $http       = new HttpServer($wsServer);
        $server     = IoServer::factory($http, $port, $ip);

        $server->run();

[WyriHaximus]

Replace that IoServer::factory with the code from it https://github.com/ratchetphp/Ratchet/blob/master/src/Ratchet/Server/IoServer.php#L67-L72 and use the desired loop.

[Mecanik]

Well this messes up my understanding of websocket server :) Thank you anyway, I will see what I can manage.

[Mecanik]

@WyriHaximus No luck. The server tries to start, but it automatically stops without any error :( Could please give me a small example ? I will take it from there and try and understand the "loop"...

[Mecanik]

@WyriHaximus I still cannot replace my event loop, I tried a lot... can you please give me an example ? I don't know what to do...

[WyriHaximus]

@Mecanik take a look at https://github.com/reactphp/event-loop/blob/master/travis-init.sh it is used be the event loop component to install loops on travis for testing.

[Mecanik]

@WyriHaximus If you are talking about "pecl event" I already installed it, and it made only a small difference from 254 connections to 1010, and it gets stuck on 1010. I honestly do not know what to do, and I need this in a production environment. I am using this "chat" example: https://github.com/pmill/php-chat

I also increased every limit there is possible on server and PHP, I am using PHP-FPM 7.1

[Mecanik]

I managed to "debug" things, it appears that the extension "ev" is not being detected at all, thus the loop factory is using "StreamSelectLoop". I am trying to install now "event" on Centos 7 with PHP 7.1 but failing so far.

[WyriHaximus]

@Mecanik did you load the .so as an extension in php.ini? When it is loaded in php php -m shows it in the list of installed extensions

[Mecanik]

@WyriHaximus Of course I did, still it`s not detected :/ is it because PHP is 7.1 ? ..

[Mecanik]

@WyriHaximus I installed ( somehow ) libevent-devel and then pecl install event and now Ratchet starts with ExtEventLoop(). Hopefully my "limit" is gone now 👯‍♂️

[Mecanik]

@WyriHaximus Final result: I have passed the 1024 with "event" because "ev" is NOT detected in PHP 7.1 by Ratchet lib.

[WyriHaximus]

@Mecanik glad to hear :+1:

[ChojinDSL]

@WyriHaximus What's the maximum number of connections you've been able to achieve so far?

[WyriHaximus]

@ChojinDSL I've stopped paying attention to that after a couple thousand

[nerdakos]

@Mecanik hello you managed to find a solution. I am still stuck at this. I use thor to test my Ratchet server and when I hit the 1020 limit no more connections are allowed. I installed Ratchet by following the website's manual and I increased all the limits in linux,nginx etc. Your answers about "loops","events" literally I don't understand what should I do. Sorry but I am a bit new on these things. Can you please explain a bit more what are the steps you followed to solve your bug?

[Mecanik]

@nerdakos - I will help you although I spent some time to understand what's actually happening.

If you go deep enough in "react/event-loop/src" you will find "Factory.php":

<?php

namespace React\EventLoop;

class Factory
{
    public static function create()
    {
        // @codeCoverageIgnoreStart
        if (function_exists('event_base_new')) {
            return new LibEventLoop();
        } elseif (class_exists('libev\EventLoop', false)) {
            return new LibEvLoop;
        } elseif (class_exists('EventBase', false)) {
            return new ExtEventLoop;
        }

        return new StreamSelectLoop();
        // @codeCoverageIgnoreEnd
    }
}

By default this will use "StreamSelectLoop" unless no other PECL extension is present in PHP.

So I created the same check for myself in order to detect if I have PECL "ev" or "event" available:

if (function_exists('event_base_new')) {
            echo "\033[0m We can use LibEventLoop!!" . PHP_EOL;
        } elseif (class_exists('libev\EventLoop', false)) {
            echo "\033[0m We can use LibEvLoop!!" . PHP_EOL;
        } elseif (class_exists('EventBase', false)) {
            echo "\033[0m We can use ExtEventLoop!!" . PHP_EOL;
        } 
        else 
        {
            echo "\033[0m We can use StreamSelectLoop!!" . PHP_EOL;
        }

And I have installed "ev" on PHP 7.1 and surprisingly it was not detecting it -_-. Then I moved installing "event" too, and it worked. The "server" will use "LibEventLoop" insteand of "StreamSelectLoop" which is limited to 1024 by default.

Few things to consider:

You will still need to increase open_files limit on the server, more exactly System resource limits. Use my code to detect the current limits and increase them accordingly:

<p>-System resource limits: <?php if (strtoupper(substr(PHP_OS, 0, 3)) === 'WIN') { echo '<span class="label label-danger">Not available on Windows!</span>'; } else { $limits = posix_getrlimit(); echo "<pre>"; print_r($limits) . PHP_EOL; echo "</pre>";}?> </p>

And of course if you are using PHP-FPM then you need to increase the limits as well, usually in php-fpm.conf ( depends on OS/PHP version ).

If you do what I said, you will be able to have many... many connections :)

[nerdakos]

@Mecanik I have to let you know that you saved me from many hours of searching for this thing. It was quite challenging to properly install libevent but in the end I managed to do it! Thanks a lot for your time!

[Mecanik]

@nerdakos - I am glad I could help man. I have spent days searching... testing... to find out this simple thing :)

[nerdakos]

@Mecanik btw after some test with libevent I realised that my setup was quite problematic. Every time I was running my tests, I was receiving the following error: "Segmentation fault (core dumped)". After some research I realised that libevent and libev are not supporting php7. My workaround was to switch to the event loop which works like a charm till now. However, more tests will make me more confident with my current setup! Again thanks a lot for your help!

[Mecanik]

@nerdakos Indeed "libevent" does not work with PHP 7, this is why I said "ev" or "event" ;)

[kelunik]

Summary stream_select or rather the underlying system call has a fixed, compiled limit of usually 1024, that means the highest file descriptor possible to use is FD 1023. That means you can have at max 1023 open connections while using stream_select. That number will usually be a bit lower, because there are other open file descriptors such as STDIN / STDOUT / STDERR or database connections, etc.

Solution Switch to an event loop implementation that doesn't use stream_select. React offers several such implementations based on extensions such as ev, some of these are PHP 7 compatible, some are not. Additionally, there's the possibility to use php-uv. There's currently no direct implementation for React, but Amp's react-adapter can be used to use Amp's UvDriver under the hood for a React application.

I think this is a good summary of the available options and this issue can be closed now.

See also https://access.redhat.com/solutions/488623.

[i3bitcoin]

I'm having the same issue with php sockets module (socket_create, socket_accept, socket_select). Is there any way to hold more than 1024 connections without recompiling php?

[kelunik]

@i3bitcoin read the comment above your's.

[i3bitcoin]

@kelunik do you have any php example of solution?

[kelunik]

@i3bitcoin You need to install an extension, there's no need for a code change as long as you don't want to use UV.

[i3bitcoin]

@kelunik sorry for my stupid questions, I'm new to this. I have php_sockets.so onboard. Which extension will solve my issue?

[kelunik]

Either ev or event from pecl.

[i3bitcoin]

Okay, I've found that my PHP Sockets server is accepting more than 1024 sockets when I run it from the root.

I'm starting it using this command. sudo -u www-data php7.0 /path/to/file/

/etc/php/7.0/fpm/php-fpm.conf
rlimit_files = 1048576

/etc/php/7.0/fpm/pool.d/www.conf
rlimit_files = 1048576
listen.backlog = 65536

/etc/security/limits.conf

• soft nproc 1048576
• hard nproc 1048576
• soft nofile 1048576
• hard nofile 1048576

Any suggestions?

[kelunik]

@i3bitcoin In that case your problem is probably ulimit -n, try increasing that.

[i3bitcoin]

su www-data --shell /bin/bash --command "ulimit -n" 1048576

it's aslo increased

[i3bitcoin]

Is there any limits for sudo command?

[amadeubarbosa]

You cannot override the 1024 connections limitation in Linux systems without recompile the kernel. See this: https://access.redhat.com/solutions/488623

It's a far more complex that some comments were mentioned ulimit. The ulimit command doesn't take effect in select for sockets when you put > 1024 values. The problem is FD_SETSIZE in libc that it's impossible to override, when you try probably the process hang, it's a undefined behaviour, actually.

Some systems as Solaris accepts FD_SETSIZE until 65536, but vanilla Linux don't.

The recomendation is to use epoll and libevent alternatives.

Further readings about the C10k problem:

• https://en.wikipedia.org/wiki/C10k_problem
• http://www.kegel.com/c10k.html

Hope to help, Amadeu Barbosa Junior

[Gemorroj]

for rhel/centos users see https://github.com/reactphp/event-loop/issues/152 php-pecl-ev extension from remi repository not supported at this time.

[AntonioDilorenzo]

hi, I'm using the socket for a web app, now that the connections are increasing, more and more often the socket hangs when the 1024 connections are reached. all the others are in the state of close_wait, I've already tried unlimit, but nothing has changed. Do you know if there is a parameter to change? thanks so much for the help!

[inri13666]

Possible solution is here

[i3bitcoin]

@AntonioDilorenzo

My solution was to use HHVM instead of PHP for sockets. It doesn't have 1024 connections limit.