wcawijngaards
commented
2 years ago This configuration does not make memory increase for me. It starts, and then the memory stays exactly the same for several minutes.
Could you run unbound with verbosity: 4, so that it logs what it is doing?
Lets calculate how much memory that is supposed to use. Every thread holds tcp buffers for incoming and outgoing. The buffer size is msg-buffer-size in unbound.conf. That defaults to about 64k, with some more bytes on it. That is 8192 66k + 16384 66k times 4 for the threads, with 66k for a bit more overhead for other data, probably slightly too high. That is 1.6G x 4 threads, about 6.4 Gb of memory.
The process starts with only a couple of hundred Mb? That must be because of some sort of VM memory system. Is swapping on? And unbound has pushed a lot of its memory to swap? As it is using it, somehow, even though you say there are no queries, the VM pressure increases, and the RSS goes up. I do not know how that VM manages the RSS.
But it is also not 15G that you say it rises to. That seems a bit more, and I wonder how that could happen. It should not cause an issue on the 32G machine. I would like to see what unbound is doing, so the log output could be helpful. You upgraded a lot of versions in one go, so I am not sure if there is some code change specifically that is causing this. Supposedly, if there are no queries, unbound is doing nothing, and thus memory can not increase. This is what it does when I start it here.
raunz
ameshkov
Unbound 1.13.1 .. 1.16.1 memory leak (complete RAM exhaustion) due to high volume of TCP FD's
After upgrading public DoT from unbound 1.12.0-1~bpo10+1 to 1.13.1-1 (debian 11 default) we ran into severe RAM exhaustion by unbound process. Starting unbound with
outgoing-num-tcp: 8192 incoming-num-tcp: 16384, the process starts up with a few hundred MB of RAM (RSS). Without any load or queries, the processes RSS starts to increase every 10 seconds almost exactly by 15168KB (the step is always between 15164 and 15172KB). This increase doesn't stop, until 99% of the hosts physical RAM is depleted - basically eating up 5.2GB of RAM every hour. On a 32GB RAM host, it took less than 6 hours to eat up all the resources. At 99% the RSS increase stops, process is still able to serve requests. As there aren't any queries being sent to unbound, the caches are empty - flushing every flusheable command with unbound-control doesn't make any difference. unbound-control reload triggers oom-killer to clean up the mess:Out of memory: Killed process 6869 (unbound) total-vm:40762004kB, anon-rss:32467148kB, file-rss:0kB, shmem-rss:0kB, UID:108 pgtables:79328kB oom_score_adj:0Detailed oom-killer output is in the attach. I built a debian deb package from latest unbound 1.16.1 release and it behaves exactly the same. Configure options are debian defaults from https://salsa.debian.org/dns-team/unboundTo reproduce Steps to reproduce the behavior: 1.Almost bare minimum configuration to trigger the issue - increase *-tcp-num and add a few threads:
2.start unbound and observe process memory usage increase every 10 seconds. no queries/clients/requests needed. 3.wait for RAM depletion (5GB per hour) and execute 'unbound-control reload' to trigger oom-killer
Expected behavior Unbound process should start up with consuming required memory for TCP FDs and not to eat up all the available RAM resources afterwards. With 1.12 we could serve ~20K concurrent DoT connections for months (until we hit tcp sockets depletion bug) and constant 60% RAM usage level, but with 1.13+ unbound we ran out RAM within hours.
System:
unbound -Voutput:Version 1.16.1 Configure line: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-option-checking --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --with-pythonmodule --with-pyunbound --enable-subnet --enable-dnstap --enable-systemd --with-libnghttp2 --with-chroot-dir= --with-dnstap-socket-path=/run/dnstap.sock --disable-rpath --with-pidfile=/run/unbound.pid --with-libevent --enable-tfo-client --with-rootkey-file=/usr/share/dns/root.key --enable-tfo-server Linked libs: libevent 2.1.12-stable (it uses epoll), OpenSSL 1.1.1n 15 Mar 2022 Linked modules: dns64 python subnetcache respip validator iterator TCP Fastopen feature available