search

mozilla-services / hindsight

Hindsight - light weight data processing skeleton
Mozilla Public License 2.0
668 stars 55 forks source link

descriptor too large for set size #129

Closed cvuillemez closed 7 years ago

cvuillemez commented 7 years ago

Hello, I'm experiencing an important issue when using Hindsight with lots of client connections to TCP input. tcp.lua plugin crashes and report this information:

1509130702213999114 [info] input.tcp detaching received: 1 msg: process_message() /usr/share/luasandbox/sandboxes/heka/input/tcp.lua:118: bad argument #1 to 'select' (descriptor too large for set size)

It appears that Hindsight is not directly in question, but the lib LuaSocket is. When Hindsight holds at least FD_SETSIZE open file descriptors, socket.select() returns an error. FD_SETSIZE is defined to 1024 by a macro in Glibc (doc).

socket.select() leverages on this macro, so when I start few hundreds of TCP clients, total of open file descriptor goes quickly above 1024 (with my settings, when no clients are connected, Hindsight still has ~250 fd, so the 1024 value is very limiting, especially for infrastructure which need to connect thousands of clients).

Maybe I could start more Hindsight instances to spread connections, but it's not a good choice, I can't be sure they will always be well distributed, and any program could flood input and makes plugin to crash.

This LuaSocket limitation is mentionned in this issue, where @diegonehab suggest to use libevent.

Regards.

cvuillemez commented 7 years ago

I opened the issue in the wrong project. I close it and re-open in lua-sandbox-extension.