Bug: configmanager throws too many open files exception

chrisvdg commented 6 years ago

Testing my scripts to setup services on the nodes in Bancadati, I also sometimes (edit: reliably deploying services on 6 + clients) get this error. Also going through a lot of clients as it's a script meant to run for the whole dc.

I also have around 315 zrobot instances saved in the configmanager for the robots on each of the nodes. I think I've been getting these errors since I've created all clients for the nodes.

js_core is at commit 28515efa893162cfc08cc4e259bc268a4c48ef4d

chrisvdg commented 6 years ago

Removed come config files of zrobot clients and that seemed to help

zaibon commented 6 years ago

@rkhamis

xmonader commented 6 years ago

openfiles.txt

producible using:

from jumpscale import j

for i in range(1000):
    zcl = j.clients.zrobot.get("r%d"%i, data={'url':'127.0.2.5', 'jwt_':"aaaaaa"})

robots = j.clients.zrobot.robots

getting as many works fine

for i in range(500):
zcl = j.clients.zrobot.get("r%d"%i)

printing robots crashes with too many open files

lsof -p PYTHONPROCESSID

COMMAND   PID USER   FD      TYPE             DEVICE SIZE/OFF     NODE NAME
....
python3 30117 root  912u     unix 0xffffa02d6a5f6400      0t0 58570949 type=STREAM
python3 30117 root  913u     unix 0xffffa02d6ca4fc00      0t0 58570000 type=STREAM
python3 30117 root  914u     unix 0xffffa02d4b0ef000      0t0 58569494 type=STREAM
python3 30117 root  915u     unix 0xffffa02d4b153c00      0t0 58569484 type=STREAM
python3 30117 root  916u     unix 0xffffa02d6a5f7000      0t0 58570112 type=STREAM
python3 30117 root  917u     unix 0xffffa02d6ef62000      0t0 58570905 type=STREAM
python3 30117 root  918u     unix 0xffffa02d4b06b000      0t0 58570923 type=STREAM
python3 30117 root  919u     unix 0xffffa02d6ca4d800      0t0 58567673 type=STREAM
python3 30117 root  920u     unix 0xffffa02d6eeaf800      0t0 58570953 type=STREAM
python3 30117 root  921u     unix 0xffffa02d4b06bc00      0t0 58570911 type=STREAM
python3 30117 root  922u     unix 0xffffa02d50e4e800      0t0 58570927 type=STREAM
python3 30117 root  923u     unix 0xffffa02d4b068400      0t0 58570915 type=STREAM
python3 30117 root  924u     unix 0xffffa02d4b06ac00      0t0 58571815 type=STREAM
python3 30117 root  925u     unix 0xffffa02db72e1000      0t0 58570033 type=STREAM
python3 30117 root  926u     unix 0xffffa02d6eeafc00      0t0 58570960 type=STREAM
python3 30117 root  927u     unix 0xffffa02d6a5f6c00      0t0 58570937 type=STREAM
python3 30117 root  928u     unix 0xffffa02db72e2000      0t0 58570130 type=STREAM
python3 30117 root  929u     unix 0xffffa02db72e2400      0t0 58570037 type=STREAM
python3 30117 root  930u     unix 0xffffa02d6eeaf400      0t0 58570968 type=STREAM
python3 30117 root  931u     unix 0xffffa02db72e0000      0t0 58570041 type=STREAM
python3 30117 root  932u     unix 0xffffa02e4698b400      0t0 58570996 type=STREAM
python3 30117 root  933u     unix 0xffffa02d6a5f4c00      0t0 58570940 type=STREAM
python3 30117 root  934u     unix 0xffffa02db72e1800      0t0 58570059 type=STREAM
python3 30117 root  935u     unix 0xffffa02db72e3800      0t0 58570045 type=STREAM
python3 30117 root  936u     unix 0xffffa02d6eeaf000      0t0 58571832 type=STREAM
python3 30117 root  937u     unix 0xffffa02db7270800      0t0 58569524 type=STREAM
python3 30117 root  938u     unix 0xffffa02d4b06b400      0t0 58571817 type=STREAM
python3 30117 root  939u     unix 0xffffa02d6ef61000      0t0 58571792 type=STREAM
python3 30117 root  940u     unix 0xffffa02db7270000      0t0 58569610 type=STREAM
python3 30117 root  941u     unix 0xffffa02d4b06a400      0t0 58571795 type=STREAM
python3 30117 root  942u     unix 0xffffa02d6eead000      0t0 58571821 type=STREAM
python3 30117 root  943u     unix 0xffffa02e4698b000      0t0 58570976 type=STREAM
python3 30117 root  944u     unix 0xffffa02d6a5f5400      0t0 58570113 type=STREAM
python3 30117 root  945u     unix 0xffffa02e4698ac00      0t0 58570980 type=STREAM
python3 30117 root  946u     unix 0xffffa02e40489000      0t0 58571002 type=STREAM
python3 30117 root  947u     unix 0xffffa02e46988400      0t0 58570984 type=STREAM
python3 30117 root  948u     unix 0xffffa02e40488400      0t0 58571892 type=STREAM
python3 30117 root  949u     unix 0xffffa02e4048a000      0t0 58571006 type=STREAM
python3 30117 root  950u     unix 0xffffa02d6eeac400      0t0 58571846 type=STREAM
python3 30117 root  951u     unix 0xffffa02e4048ac00      0t0 58571010 type=STREAM
python3 30117 root  952u     unix 0xffffa02e4048a800      0t0 58570185 type=STREAM
python3 30117 root  953u     unix 0xffffa02db72e3c00      0t0 58570097 type=STREAM
python3 30117 root  954u     unix 0xffffa02d4b608800      0t0 58571852 type=STREAM
python3 30117 root  955u     unix 0xffffa02d6eeae800      0t0 58571827 type=STREAM
python3 30117 root  956u     unix 0xffffa02db72e3000      0t0 58571895 type=STREAM
python3 30117 root  957u     unix 0xffffa02d5d5f1000      0t0 58570105 type=STREAM
python3 30117 root  958u     unix 0xffffa02d6a5f7400      0t0 58570114 type=STREAM
python3 30117 root  959u     unix 0xffffa02db7272800      0t0 58570108 type=STREAM
python3 30117 root  960u     unix 0xffffa02d4b7f6c00      0t0 58570161 type=STREAM
python3 30117 root  961u     unix 0xffffa02e40489400      0t0 58569561 type=STREAM
python3 30117 root  962u     unix 0xffffa02d6ca4dc00      0t0 58570139 type=STREAM
python3 30117 root  963u     unix 0xffffa02d4b068800      0t0 58570118 type=STREAM
python3 30117 root  964u     unix 0xffffa02e40403000      0t0 58569688 type=STREAM
python3 30117 root  965u     unix 0xffffa02d6ef61800      0t0 58571872 type=STREAM
python3 30117 root  966u     unix 0xffffa02da6c41c00      0t0 58571899 type=STREAM
python3 30117 root  967u     unix 0xffffa02e40488000      0t0 58569568 type=STREAM
python3 30117 root  968u     unix 0xffffa02e4048e000      0t0 58571028 type=STREAM
python3 30117 root  969u     unix 0xffffa02e46988c00      0t0 58569572 type=STREAM
python3 30117 root  970u     unix 0xffffa02d6ef63000      0t0 58571904 type=STREAM
python3 30117 root  971u     unix 0xffffa02e4698a000      0t0 58569574 type=STREAM
python3 30117 root  972u     unix 0xffffa02d6a5ea400      0t0 58569634 type=STREAM
python3 30117 root  973u     unix 0xffffa02d4b7f4400      0t0 58570146 type=STREAM
python3 30117 root  974u     unix 0xffffa02e4048c800      0t0 58571036 type=STREAM
python3 30117 root  975u     unix 0xffffa02d4b7f4c00      0t0 58570150 type=STREAM
python3 30117 root  976u     unix 0xffffa02e4048f800      0t0 58572807 type=STREAM
python3 30117 root  977u     unix 0xffffa02db7264c00      0t0 58571911 type=STREAM
python3 30117 root  978u     unix 0xffffa02d4b7f5400      0t0 58570162 type=STREAM
python3 30117 root  979u     unix 0xffffa02db7265400      0t0 58571915 type=STREAM
python3 30117 root  980u     unix 0xffffa02d6efd0c00      0t0 58571071 type=STREAM
python3 30117 root  981u     unix 0xffffa02db7264800      0t0 58571920 type=STREAM
python3 30117 root  982u     unix 0xffffa02d4b7f5000      0t0 58570164 type=STREAM
python3 30117 root  983u     unix 0xffffa02d6a5de400      0t0 58569605 type=STREAM
python3 30117 root  984u     unix 0xffffa02e40402000      0t0 58569692 type=STREAM
python3 30117 root  985u     unix 0xffffa02d6a5eb400      0t0 58569619 type=STREAM
python3 30117 root  986u     unix 0xffffa02d4b60b800      0t0 58570204 type=STREAM
python3 30117 root  987u     unix 0xffffa02d4b7f6400      0t0 58570169 type=STREAM
python3 30117 root  989u     unix 0xffffa02d6efd1800      0t0 58571053 type=STREAM
python3 30117 root  990u     unix 0xffffa02d6a5e8000      0t0 58569639 type=STREAM
python3 30117 root  991u     unix 0xffffa02d6efd3400      0t0 58571057 type=STREAM
python3 30117 root  992u     unix 0xffffa02db7267c00      0t0 58571940 type=STREAM
python3 30117 root  993u     unix 0xffffa02d6efd0800      0t0 58571062 type=STREAM
python3 30117 root  994u     unix 0xffffa02d6efd2400      0t0 58571076 type=STREAM
python3 30117 root  995u     unix 0xffffa02d4b7f7000      0t0 58570180 type=STREAM
python3 30117 root  997u     unix 0xffffa02d6a5dd800      0t0 58569652 type=STREAM
python3 30117 root  998u     unix 0xffffa02e4048e400      0t0 58571085 type=STREAM
python3 30117 root  999u     unix 0xffffa02d6a5f7800      0t0 58569660 type=STREAM
python3 30117 root 1001u     unix 0xffffa02e40401c00      0t0 58569668 type=STREAM
python3 30117 root 1002u     unix 0xffffa02d5d5f3c00      0t0 58571945 type=STREAM
python3 30117 root 1003u     unix 0xffffa02e40400400      0t0 58569676 type=STREAM
python3 30117 root 1005u     unix 0xffffa02d6efd0400      0t0 58571077 type=STREAM
python3 30117 root 1006u     unix 0xffffa02e4048d800      0t0 58571095 type=STREAM
python3 30117 root 1007u     unix 0xffffa02db7266400      0t0 58571928 type=STREAM
python3 30117 root 1009u     unix 0xffffa02e4048d400      0t0 58571100 type=STREAM
python3 30117 root 1011u     unix 0xffffa02e40462400      0t0 58571104 type=STREAM
python3 30117 root 1013u     unix 0xffffa02e40489800      0t0 58571946 type=STREAM
python3 30117 root 1015u     unix 0xffffa02e4046a000      0t0 58571950 type=STREAM
python3 30117 root 1017u     unix 0xffffa02e4046b800      0t0 58571954 type=STREAM
python3 30117 root 1019u     unix 0xffffa02d4b7f6800      0t0 58572803 type=STREAM
python3 30117 root 1020u     unix 0xffffa02d5d414000      0t0 58591502 type=STREAM
...
...
MORE OUTPUT OMITTED

So i think it maybe a problem of getting all the clients to robots at the same time issuing too many connections on the ssh agent socket.

when using less heavy repr works fine "postpones" the problem

def __repr__(self):

    return "emptry repr.."
    # robots = {}
    # for instance in j.clients.zrobot.list():
    #     robots[instance] = ZeroRobotManager(instance)
    # return str(robots)

zaibon commented 6 years ago

@rkhamis @xmonader: https://github.com/threefoldtech/0-robot/commit/fad034d075c69a20aae170a83418692a2fc41642

threefoldtecharchive / jumpscale9_core

Bug: configmanager throws too many open files exception #136