lynckia / licode

Open Source Communication Provider based on WebRTC and Cloud technologies
http://lynckia.com/licode
MIT License
3.1k stars 1.01k forks source link

Erizo Crash: eventfd_select_interrupter: Too many open files #102

Closed johnwarden closed 10 years ago

johnwarden commented 10 years ago

I get this error in Erizo fairly consistently whenever a room reaches 10+ publishers/subscribers at the same time:

  terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::system::system_error> >'
    what():  eventfd_select_interrupter: Too many open files
  /home/ubuntu/licode/scripts/erizo_controller_service.sh: line 22: 12594 Aborted                 (core dumped) node erizoController.js

FYI erizo_controller_service.sh is just a wrapper that runs node erizoController.js with some restart logic.

peili commented 10 years ago

I get a similar error whenever it reaches 10+ publishers/subscribers at the same time,

This makes production usage useless at the moment.

terminate called after throwing an instance of >'boost::exception_detail::clone_implboost::exception_detail::error_info_injector >' what(): epoll: Too many open files

AndreasKalva commented 10 years ago

I have the same issue.

Please help ASAP. This is essential for us!!!

Thank you

jcague commented 10 years ago

Hi all,

This problem can be resolved by increasing the limit of open files by each process in Linux. There are several ways to do this. For example, we do this in the running script of Licode's test page:

limit nofile 32768 32768 limit core unlimited unlimited

This script is within /etc/init/ directory, so it's an upstart script we use to start tasks like Erizo Controller. But you can find similar solutions in other scripting toolkits.

Cheers, Javier.

On 21 April 2014 20:18, AndreasKalva notifications@github.com wrote:

I have the same issue.

Please help ASAP. This is essential for us!!!

Thank you

— Reply to this email directly or view it on GitHubhttps://github.com/ging/licode/issues/102#issuecomment-40960277 .

peili commented 10 years ago

Hi Javier,

thanks for your reply. Why does Licode need so many file descriptors? I know the number of sockets is proportional to the square of the number of connected users of the room. But the default value of 1024 should be high enough?

If you don't mind It would be helpful to publish your upstart script also in github.

Cheers,

peili

johnwarden commented 10 years ago

I use supervisor to launch erizo controller, so couldn't do the above (Ubuntu precise).

But I was able to increase the limit by calling 'ulimit' in initErizo_controller.sh before 'node erizoController.js'

ulimit -n 4096 # nofile limit
ulimit -c unlimited # core limit

I don't set the nofile limit to more than 4096, because the hard nofile limit is 4096 on my system, and changing this requires some tricker system admin work.

After making this change and restarting erizo, you can check the actual max open files limit by looking in

/proc/$PID/limits

Here's a handy shortcut:

cat /proc/`ps -C 'node erizoController.jz' -o pid= | awk '{$1=$1}1'`/limits
jcague commented 10 years ago

With the info given by John it's solved. Anyway in the new code it will not be such a problem for multiconference rooms because we will create new processes for every publisher, so they will not share the limit on the number of open files.