grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
24.01k stars 3.46k forks source link

Why does promtail's child process keep growing? #15067

Open Nemenisi opened 3 days ago

Nemenisi commented 3 days ago

I started promtail_v3.2.0 with docker. But the longer promtail runs, the more child processes it has. When the pids.max is reached, the system stops executing promtail. Is there any way I can limit it from increasing?

background:

pstree -p 557123 conmon(557123)─┬─promtail(557135)─┬─{promtail}(557147)                     │ ├─{promtail}(557148)                     │ ├─{promtail}(557149)                     │ ├─{promtail}(557150)                     │ ├─{promtail}(557151)                     ......                    

The promtail main process pid is 557135. It has a number of child processes. Let's take a look at the detailed process information.

top -H -p 557135 top - 17:29:59 up 480 days, 1:34, 0 users, load average: 1.45, 1.63, 1.63 Threads: 143 total, 0 running, 143 sleeping, 0 stopped, 0 zombie %Cpu(s): 0.4 us, 0.4 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.2 hi, 0.0 si, 0.0 st MiB Mem : 515266.3 total, 490612.8 free, 6414.5 used, 18239.1 buff/cache MiB Swap: 614400.0 total, 614400.0 free, 0.0 used. 500362.1 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 557308 root 20 0 12.1g 970504 58756 S 6.7 0.2 0:13.87 promtail 563628 root 20 0 12.1g 970504 58756 S 6.7 0.2 0:22.11 promtail 557135 root 20 0 12.1g 970504 58756 S 0.0 0.2 0:00.01 promtail 557147 root 20 0 12.1g 970504 58756 S 0.0 0.2 0:17.60 promtail 557151 root 20 0 12.1g 970504 58756 S 0.0 0.2 0:00.00 promtail ... 557180 root 20 0 12.1g 970504 58756 S 0.0 0.2 0:19.69 promtail 557181 root 20 0 12.1g 970504 58756 S 0.0 0.2 0:33.38 promtail 557183 root 20 0 12.1g 970504 58756 S 0.0 0.2 0:34.51 promtail

Why are there so many sleeping child processes? And will there be more and more? How can I limit its growth?