[Question][Help req] bunch of runc threads, high cpu load

[marech]

Hey!

Im trying to debug high load on servers and somehow it lead me to runc.

So the problem is that when i run forkstat command on server i see a lot of following output and it does not stop:

Time     Event   PID Info   Duration Process
13:58:16 clone 19966 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19970 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19970      0    0.028 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19968      0    0.029 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19967      0    0.029 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19969      0    0.029 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19966      0    0.036 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 fork  27527 parent          containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
13:58:16 fork  19971  child          containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
13:58:16 exec  19971                 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19971 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19972 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19971 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19973 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19971 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19974 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19971 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19975 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19971 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19976 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19974      0    0.027 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19975      0    0.027 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19973      0    0.028 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19971      0    0.035 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19972      0    0.028 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19976      0    0.006 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 fork  27527 parent          containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
13:58:16 fork  19977  child          containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
13:58:16 exec  19977                 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19977 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19978 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19977 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19979 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19977 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19980 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19977 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19981 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19977 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19982 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19977      0    0.031 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19981      0    0.025 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19979      0    0.026 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19980      0    0.026 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19978      0    0.026 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19982      0    0.008 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 fork  27527 parent          containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
13:58:16 fork  19983  child          containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
13:58:16 exec  19983                 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19983 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19984 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19983 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19985 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19983 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19986 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19983 parent          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 clone 19987 thread          runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19986      0    0.028 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19987      0    0.028 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19985      0    0.028 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19984      0    0.028 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053
13:58:16 exit  19983      0    0.036 runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053/log.json --log-format json state 4a2ad302bce4fd2d41665d6ab76c380c803b94ee40bf444d904783e46eeba053

Could someone give some insights what this means or how to read, interpret it? Like something in containers are forking a lot?

The whole story is helm redis-ha chart related, when i deploy redis-ha to kubernetes cluster.

[alitvak69]

I will take an uneducated guess that runc is failing to fully start the container and hangs. I have seen with podman running ceph containers runc fails to open /dev/stderr on ubuntu 18.04.2 and hangs. Systemd auto restarts podman on failure multiple times leaving runc processes hanging and making system load higher to the point that systemctl daemon-reload takes 10 - 15 sec to compelte if not longer.

[marech]

Weird part is that nothing crashes, like all containers are up and running. Good point about OS, will try meanwhile 16.04 ubuntu or any other suggestions which way i should dig? :)

[zq-david-wang]

It seems to me that your logs was filtered showing only lines contains runc. If that is the case, you'd better check logs around to find out what the first command attaching to runc.

[marech]

Tried on ubuntu 16.04, sadly same problem.

@zq-david-wang yep logs were filtered too much, here i gathered little bit more forkstat.log

That sh /probes/readiness.sh 6379 looks suspicious.

How exactly spawning processes in existing container works? Something like : kubelet(or i guess any other script which interacts with runc) wants to run script -> delegates it to runc -> runc then attaches to existing container -> spawns and executes script -> script finishes -> runc detaches from container?

[zq-david-wang]

kubelet iiveness/readiness probe period is tunable, if you suspect that is the problem, you can disable liveness/readiness probe to verify.

[marech]

yep liveness/readiness probes are the root cause, disabled them and load dropped by 10-15%. forkstat now is calm, does not spam about runc

Thanks for help!