Grandpa dies from file descriptor exhaustion

[bh9]

Bug Report

iRODS Version, OS and Version

4.2.7 Ubuntu 18.04 running 16.04 binaries

What did you try to do?

High parallel load from igets on 2 consumers exhaused the file descriptors available to the original (grandpa) irodsServer process Once the file descriptors are exhausted, the server process can not function until it is restarted, and any connection attempts will receive a USER_SOCK_CONNECT_TIMEDOUT.

Workaround

We plan to work around this issue by bumping the number of available file descriptors from the default 1024 to a more sensible 131072 In systemd, this is done by adding the following to the irods service file

LimitNOFILE=131072

If a systemd service file is not in use, it should be sufficient to edit /etc/security/limits.conf to include

*     hard     nofile     131072
*     soft     nofile     131072

[trel]

Need to determine whether

• reaching that 1024 is itself a bug (we're leaking file descriptors), or
• we should document a higher value to be used for an iRODS server behaving as expected, or
• we should make the irodsServer handle hitting that limit more gracefully.

Perhaps we should do the last one, regardless.