william-stearns
commented
3 weeks ago This issue caused Zeek, the above-mentioned user of libkqueue, to grow from <300M of memory used to 4.2GB of resident ram used (16GB virtual) when run on a system that has this large number of file descriptors (which appears to be RHEL 9/centos stream 9/alma 9/ and rocky 9. Is it possible to adjust the size of these arrays to avoid having zeek allocate this much ram, triggering the oom-killer?
We discovered a problem recently where the monitoring thread spawned by libkqueue was using a very large amount of memory. After a bit of digging, we realized that it's allocating two arrays of
ints that are the size of the maximum number of file descriptors on the system. The system we discovered this on is a Rocky 9 system, with a docker container running in containerd. In this environment, containerd sets the maximum number of descriptors to 1073741816. libkqueue then allocates about 8GB of memory.While it is possible to tune the container down to use fewer descriptors, this seems like an upstream problem in libkqueue itself.