fuuuuu-Server crashes

[logsol]

After a while of running on the server, the application breaks.

I googled "Fatal error in ../src/platform-linux.cc, line 782" and found that it means: "That's V8 aborting because you hit RLIMIT_NPROC (ulimit -u) or the system-wide process/thread limit."

We should look into why this is happening, if it is a misconfiguration of the server or if we are somehow creating lots of threads. Maybe run it on a different server to see, if it reoccurs.

I don't know if the other errors have anything to do with this. it looks like a socket issue, maybe the connection is lost or something?

Here is the complete error log:

Error: channel closed
    at ChildProcess.target.send (child_process.js:402:26)
    at PipeToChannel.send (/home/chuck/development/source/app/Server/PipeToChannel.js:38:13)
    at User.onDisconnect (/home/chuck/development/source/app/Server/User.js:48:26)
    at Socket.EventEmitter.emit [as $emit] (events.js:95:17)
    at Socket.onDisconnect (/home/chuck/development/source/node_modules/socket.io/lib/socket.js:153:10)
    at SocketNamespace.handleDisconnect (/home/chuck/development/source/node_modules/socket.io/lib/namespace.js:229:46)
    at Manager.onClientDisconnect (/home/chuck/development/source/node_modules/socket.io/lib/manager.js:481:27)
    at WebSocket.Transport.end (/home/chuck/development/source/node_modules/socket.io/lib/transport.js:467:20)
    at Parser.<anonymous> (/home/chuck/development/source/node_modules/socket.io/lib/transports/websocket/hybi-16.js:52:10)
    at Parser.EventEmitter.emit (events.js:92:17)

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:402:26)
    at PipeToChannel.send (/home/chuck/development/source/app/Server/PipeToChannel.js:38:13)
    at User.onDisconnect (/home/chuck/development/source/app/Server/User.js:48:26)
    at Socket.EventEmitter.emit [as $emit] (events.js:95:17)
    at Socket.onDisconnect (/home/chuck/development/source/node_modules/socket.io/lib/socket.js:153:10)
    at SocketNamespace.handleDisconnect (/home/chuck/development/source/node_modules/socket.io/lib/namespace.js:229:46)
    at Manager.onClientDisconnect (/home/chuck/development/source/node_modules/socket.io/lib/manager.js:481:27)
    at WebSocket.Transport.end (/home/chuck/development/source/node_modules/socket.io/lib/transport.js:467:20)
    at Parser.<anonymous> (/home/chuck/development/source/node_modules/socket.io/lib/transports/websocket/hybi-16.js:52:10)
    at Parser.EventEmitter.emit (events.js:92:17)
events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:402:26)
    at PipeToChannel.send (/home/chuck/development/source/app/Server/PipeToChannel.js:38:13)
    at User.onJoin (/home/chuck/development/source/app/Server/User.js:67:26)
    at Object.Helper.applyCommand (/home/chuck/development/source/app/Lib/Utilities/Protocol/Helper.js:36:32)
    at User.onMessage (/home/chuck/development/source/app/Server/User.js:39:24)
    at Socket.EventEmitter.emit [as $emit] (events.js:95:17)
    at SocketNamespace.handlePacket (/home/chuck/development/source/node_modules/socket.io/lib/namespace.js:353:20)
    at Manager.onClientMessage (/home/chuck/development/source/node_modules/socket.io/lib/manager.js:469:38)
    at WebSocket.Transport.onMessage (/home/chuck/development/source/node_modules/socket.io/lib/transport.js:387:20)
    at Parser.<anonymous> (/home/chuck/development/source/node_modules/socket.io/lib/transports/websocket/hybi-16.js:39:10)

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:402:26)
    at PipeToChannel.send (/home/chuck/development/source/app/Server/PipeToChannel.js:38:13)
    at User.onJoin (/home/chuck/development/source/app/Server/User.js:67:26)
    at Object.Helper.applyCommand (/home/chuck/development/source/app/Lib/Utilities/Protocol/Helper.js:36:32)
    at User.onMessage (/home/chuck/development/source/app/Server/User.js:39:24)
    at Socket.EventEmitter.emit [as $emit] (events.js:95:17)
    at SocketNamespace.handlePacket (/home/chuck/development/source/node_modules/socket.io/lib/namespace.js:353:20)
    at Manager.onClientMessage (/home/chuck/development/source/node_modules/socket.io/lib/manager.js:469:38)
    at WebSocket.Transport.onMessage (/home/chuck/development/source/node_modules/socket.io/lib/transport.js:387:20)
    at Parser.<anonymous> (/home/chuck/development/source/node_modules/socket.io/lib/transports/websocket/hybi-16.js:39:10)

#
# Fatal error in ../src/platform-linux.cc, line 782
# CHECK_EQ(0, result) failed
#   Expected: 0
#   Found: 11
#

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:402:26)
    at PipeToChannel.send (/home/chuck/development/source/app/Server/PipeToChannel.js:38:13)
    at User.onJoin (/home/chuck/development/source/app/Server/User.js:67:26)
    at Object.Helper.applyCommand (/home/chuck/development/source/app/Lib/Utilities/Protocol/Helper.js:36:32)
    at User.onMessage (/home/chuck/development/source/app/Server/User.js:39:24)
    at Socket.EventEmitter.emit [as $emit] (events.js:95:17)
    at SocketNamespace.handlePacket (/home/chuck/development/source/node_modules/socket.io/lib/namespace.js:353:20)
    at Manager.onClientMessage (/home/chuck/development/source/node_modules/socket.io/lib/manager.js:469:38)
    at WebSocket.Transport.onMessage (/home/chuck/development/source/node_modules/socket.io/lib/transport.js:387:20)
    at Parser.<anonymous> (/home/chuck/development/source/node_modules/socket.io/lib/transports/websocket/hybi-16.js:39:10)

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: channel closed
    at ChildProcess.target.send (child_process.js:402:26)
    at PipeToChannel.send (/home/chuck/development/source/app/Server/PipeToChannel.js:38:13)
    at User.onDisconnect (/home/chuck/development/source/app/Server/User.js:48:26)
    at Socket.EventEmitter.emit [as $emit] (events.js:95:17)
    at Socket.onDisconnect (/home/chuck/development/source/node_modules/socket.io/lib/socket.js:153:10)
    at SocketNamespace.handleDisconnect (/home/chuck/development/source/node_modules/socket.io/lib/namespace.js:229:46)
    at Manager.onClientDisconnect (/home/chuck/development/source/node_modules/socket.io/lib/manager.js:481:27)
    at WebSocket.Transport.end (/home/chuck/development/source/node_modules/socket.io/lib/transport.js:467:20)
    at Parser.<anonymous> (/home/chuck/development/source/node_modules/socket.io/lib/transports/websocket/hybi-16.js:52:10)
    at Parser.EventEmitter.emit (events.js:92:17)

[logsol]

we found the occurance of CHECK_EQ function in the ../src/platform-linux.cc script inside V8.

 int result = pthread_create(&data_->thread_, attr_ptr, ThreadEntry, this);
  CHECK_EQ(0, result);
  ASSERT(data_->thread_ != kNoThread);

11 means: The system lacked the necessary resources to create another thread, or the system-imposed limit on the total number of threads in a process PTHREAD_THREADS_MAX would be exceeded (http://stackoverflow.com/questions/7038586/return-code-from-pthread-create-is-11)

So far we added a cronjob, that counts the number or threads of chuck and of the system every hour, so that we can see if at some point there really are too many threads open (see /home/jeena/count-threads.log)

#!/bin/sh

SERVERS=`ps axH | grep server | grep -v grep | wc -l`
CLIENTS=`ps axH | grep server | grep -v grep | wc -l`
THREADS=`ps axH | wc -l`
DATE=`date`

echo "$DATE:\t$SERVERS\t$CLIENTS\t$THREADS" >> /home/jeena/count-threads.log

[jeena]

Under Alpha 2 this happened just after you joined a room.