search

icecc / icecream

Distributed compiler with a central scheduler to share build load
GNU General Public License v2.0
1.58k stars 248 forks source link

accept error(Error: Too many open files) #588

Open dwrobel opened 3 years ago

dwrobel commented 3 years ago

I installed the 1.4rc1 version on five x86_64 machines and started to compile linux kernel sources using: make defconfig; make -j all

On the localhost machine I've observed Error: Too many open files):

$ systemctl status iceccd
● iceccd.service - Icecream Distributed Compiler
     Loaded: loaded (/usr/lib/systemd/system/iceccd.service; disabled; vendor preset: disabled)
     Active: active (running) since Sat 2021-07-03 16:52:18 CEST; 9min ago
   Main PID: 515279 (iceccd)
      Tasks: 1 (limit: 18884)
     Memory: 31.5M
        CPU: 8min 7.496s
     CGroup: /system.slice/iceccd.service
             └─515279 /usr/sbin/iceccd -u icecc -b /var/cache/icecream -v -m 6

Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept error(Error: Too many open files)
Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept failed:(Error: Too many open files)
Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept error(Error: Too many open files)
Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept failed:(Error: Too many open files)
Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept error(Error: Too many open files)
Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept failed:(Error: Too many open files)
Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept error(Error: Too many open files)
Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept failed:(Error: Too many open files)
Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept error(Error: Too many open files)
Jul 03 17:00:54 dell iceccd[515279]: [515279] 2021-07-03 17:00:54: accept failed:(Error: Too many open files)

Looks like it reached the limit:

$ sudo -u icecc ulimit -n
1024
$ sudo lsof -p 515279 | wc -l
1052

The /var/run/icecc/iceccd.socket seemed to be opened too many times:

$ sudo lsof -p 515279 | head -n 100
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
      Output information may be incomplete.
lsof: WARNING: can't stat() fuse.portal file system /run/user/1000/doc
      Output information may be incomplete.
COMMAND    PID  USER   FD   TYPE             DEVICE SIZE/OFF       NODE NAME
iceccd  515279 icecc  cwd    DIR               0,33      184        256 /
iceccd  515279 icecc  rtd    DIR               0,33      184        256 /
iceccd  515279 icecc  txt    REG               0,33   192056    1720135 /usr/sbin/iceccd
iceccd  515279 icecc  mem    REG               0,31             1720135 /usr/sbin/iceccd (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1493670 /usr/lib64/libSegFault.so (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1712713 /var/lib/sss/mc/passwd (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1524273 /usr/lib64/libnss_sss.so.2 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               30588 /usr/lib64/libattr.so.1.1.2501 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1493687 /usr/lib64/libpthread-2.33.so (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1136906 /usr/lib64/libxml2.so.2.9.12 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               31627 /usr/lib64/libz.so.1.2.11 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               30647 /usr/lib64/libbz2.so.1.0.8 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               31092 /usr/lib64/liblz4.so.1.9.3 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               31093 /usr/lib64/liblzma.so.5.2.5 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               30556 /usr/lib64/libacl.so.1.1.2301 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               30709 /usr/lib64/libcrypto.so.1.1.1k (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1493675 /usr/lib64/libdl-2.33.so (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31              885978 /usr/lib64/libzstd.so.1.5.0 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               31094 /usr/lib64/liblzo2.so.2.0.0 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1493673 /usr/lib64/libc-2.33.so (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1135980 /usr/lib64/libgcc_s-11-20210531.so.1 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1493677 /usr/lib64/libm-2.33.so (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1136888 /usr/lib64/libstdc++.so.6.0.29 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               30574 /usr/lib64/libarchive.so.13.5.1 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31               30660 /usr/lib64/libcap-ng.so.0.0.0 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1720123 /usr/lib64/libicecc.so.0.0.0 (path dev=0,33)
iceccd  515279 icecc  mem    REG               0,31             1493666 /usr/lib64/ld-2.33.so (path dev=0,33)
iceccd  515279 icecc    0r   CHR                1,3      0t0          4 /dev/null
iceccd  515279 icecc    1u  unix 0x00000000fc83c15e      0t0     826259 type=STREAM (CONNECTED)
iceccd  515279 icecc    2u  unix 0x00000000fc83c15e      0t0     826259 type=STREAM (CONNECTED)
iceccd  515279 icecc    3w   CHR                1,3      0t0          4 /dev/null
iceccd  515279 icecc    4r   REG               0,33  9253600    1712713 /var/lib/sss/mc/passwd
iceccd  515279 icecc    5u  IPv4             840707      0t0        TCP *:10245 (LISTEN)
iceccd  515279 icecc    6u  unix 0x00000000f46a43ac      0t0     840708 /var/run/icecc/iceccd.socket type=STREAM (LISTEN)
iceccd  515279 icecc    7u  IPv4             841055      0t0        TCP dell:51266->_gateway:ultraseek-http (ESTABLISHED)
iceccd  515279 icecc    8r   REG               0,21        0 4026532042 /proc/stat
iceccd  515279 icecc    9r   REG               0,21        0 4026532041 /proc/meminfo
iceccd  515279 icecc   10u  unix 0x0000000011dc5151      0t0     847964 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   11u  unix 0x0000000031aca8bf      0t0     847970 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   12u  unix 0x00000000c2c30c72      0t0     846218 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   13u  unix 0x00000000658f0518      0t0     845330 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   14u  unix 0x00000000c30d8534      0t0     847974 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   15u  unix 0x0000000048ecbb2a      0t0     845615 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   16u  unix 0x0000000021a14e48      0t0     846223 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   17u  unix 0x000000000d2a1012      0t0     846240 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   18u  unix 0x000000008976564b      0t0     846245 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   19u  unix 0x0000000088c1766e      0t0     845616 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   20u  unix 0x00000000fa72e027      0t0     846248 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   21u  unix 0x000000001c9828bf      0t0     846252 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   22u  unix 0x00000000f47443af      0t0     846253 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   23u  unix 0x00000000b1752870      0t0     845202 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   24u  unix 0x0000000042011189      0t0     845209 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   25u  unix 0x000000001f1c56c2      0t0     845215 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   26u  unix 0x0000000084438a96      0t0     845220 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   27u  unix 0x0000000011c121b9      0t0     847087 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   28u  unix 0x00000000bbcd687e      0t0     845464 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   29u  unix 0x00000000de4b8ffe      0t0     847090 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   30u  unix 0x00000000315b11c7      0t0     847094 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   31u  unix 0x000000008109cc44      0t0     846269 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   32u  unix 0x00000000a7da7576      0t0     846270 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   33u  unix 0x0000000048e05c46      0t0     846273 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   34u  unix 0x000000009b672189      0t0     846274 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   35u  unix 0x00000000081b8089      0t0     846278 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   36u  unix 0x00000000c45ac4dc      0t0     847101 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   37u  unix 0x00000000347599e5      0t0     847102 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   38u  unix 0x00000000025efa84      0t0     845480 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   39u  unix 0x00000000bfc6a3d2      0t0     849052 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   40u  unix 0x0000000086f5f057      0t0     846284 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   41u  unix 0x00000000b35a1ae3      0t0     845621 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   42u  unix 0x000000001cf1c345      0t0     846286 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   43u  unix 0x0000000009704f09      0t0     846287 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   44u  unix 0x00000000584d20da      0t0     846289 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   45u  unix 0x00000000c58c3801      0t0     846290 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   46u  unix 0x000000004c737ab6      0t0     845471 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   47u  unix 0x00000000a0d6ddbf      0t0     846295 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   48u  unix 0x000000005a75efb9      0t0     847111 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   49u  unix 0x00000000d631ac8d      0t0     847112 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   50u  unix 0x0000000030394443      0t0     845622 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   51u  unix 0x000000006caf6281      0t0     845623 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   52u  unix 0x000000002d273880      0t0     847123 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   53u  unix 0x00000000220504ff      0t0     845624 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   54u  unix 0x000000005366f5e3      0t0     845237 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   55u  unix 0x000000008990b93c      0t0     845238 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   56u  unix 0x00000000d8aaad1c      0t0     845239 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   57u  unix 0x00000000ea122c72      0t0     845240 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   58u  unix 0x0000000077c53840      0t0     845627 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   59u  unix 0x000000001b53eb6b      0t0     845243 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   60u  unix 0x00000000f048a5ed      0t0     845244 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   61u  unix 0x00000000dafa52dd      0t0     845246 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   62u  unix 0x00000000c888068e      0t0     845247 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   63u  unix 0x0000000058f7f064      0t0     845248 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   64u  unix 0x0000000044500a8c      0t0     845249 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   65u  unix 0x00000000785143bd      0t0     845250 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   66u  unix 0x00000000499526d4      0t0     845251 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   67u  unix 0x000000008ae5cb52      0t0     845252 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   68u  unix 0x00000000b2f37f36      0t0     845253 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   69u  unix 0x00000000dd82b19a      0t0     845254 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   70u  unix 0x00000000e50d1b77      0t0     845255 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)
iceccd  515279 icecc   71u  unix 0x0000000012c05718      0t0     845702 /var/run/icecc/iceccd.socket type=STREAM (CONNECTED)

Starting from that any further compilation seems to get stuck and you need to check all machines and execute systemctl restart iceccd on the affected one to get it working again.