Router overloading after 2k concurrent users

Hi,

@mbonneau - Duplicated this thread with more information as google groups app doesn't let me reply to the post.

I have Thruway/Ratchet implementation in our production environment where 4-5k concurrent users logging in during the day (thanks for the great library).

After so many connections, router is going to idle state with over 95% cpu usage.

When i checked the logs, I could see last entry was "RatchetTransportProvider::onOpen" and hanging from there. Router is not responding to both new and existing connections.

Please can you help and advise asap as it is impacting our users ?

FYI.

EventLoop is "Libevent"
I have checked the supervisor config and ulimit on the servers, and please find the output below:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 197543
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 131072
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Limit configuration of Router process is below

Limit                     Soft Limit           Hard Limit           Units    
Max cpu time              unlimited            unlimited            seconds  
Max file size             unlimited            unlimited            bytes    
Max data size             unlimited            unlimited            bytes    
Max stack size            10485760             unlimited            bytes    
Max core file size        0                    unlimited            bytes    
Max resident set          unlimited            unlimited            bytes    
Max processes             4096                 4096                 processes
Max open files            131072               131072               files    
Max locked memory         65536                65536                bytes    
Max address space         unlimited            unlimited            bytes    
Max file locks            unlimited            unlimited            locks    
Max pending signals       197543               197543               signals  
Max msgqueue size         819200               819200               bytes    
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

Composer Info

cboden/ratchet                       v0.3.6
ratchet/pawl                         v0.1.2
react/cache                          v0.4.1
react/child-process                  v0.4.3
react/dns                            v0.4.6
react/event-loop                     v0.4.2
react/promise                        v2.5.0
react/promise-timer                  v1.1.1
react/socket                         v0.4.6
react/socket-client                  v0.4.6
react/stomp                          dev-emit-end 9eebc9e        
react/stream                         v0.4.6
voryx/thruway                        0.4.2
voryx/thruway-bundle                 0.1.0

Platform Info


Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
CPU(s):                8
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             8
Model name:            Intel(R) Xeon(R) CPU E7- 8837 2.67GHz

ratchetphp / Ratchet

Router overloading after 2k concurrent users #591