search

Granulate / gprofiler

gProfiler is a system-wide profiler, combining multiple sampling profilers to produce unified visualization of what your CPU is spending time on.
https://profiler.granulate.io
Apache License 2.0
756 stars 55 forks source link

perf: memory leak #528

Open Jongy opened 2 years ago

Jongy commented 2 years ago

Occasionally I'm seeing perf processes whose memory continues growing forever. Here's one example for /proc/pid/status of such a perf - it shows that anonymous memory takes up most of the space (700mb). This perf process is at least 2 weeks old.

ame:    perf
Umask:  0022
State:  S (sleeping)
Tgid:   3966
Ngid:   0
Pid:    3966
PPid:   3825
TracerPid:  0
Uid:    0   0   0   0
Gid:    0   0   0   0
FDSize: 64
Groups: 0 1001 
NStgid: 3966
NSpid:  3966
NSpgid: 3966
NSsid:  3665
VmPeak:   732316 kB
VmSize:   732208 kB
VmLck:         0 kB
VmPin:      5136 kB
VmHWM:    707368 kB
VmRSS:    707368 kB
RssAnon:      702548 kB
RssFile:        4820 kB
RssShmem:          0 kB
VmData:   712220 kB
VmStk:       132 kB
VmExe:     11512 kB
VmLib:         4 kB
VmPTE:      1452 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
THP_enabled:    1
Threads:    2
SigQ:   0/15540
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000008000000
SigIgn: 0000000000000004
SigCgt: 0000000180014c02
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp:    0
Speculation_Store_Bypass:   thread vulnerable
Cpus_allowed:   3
Cpus_allowed_list:  0-1
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:  0
voluntary_ctxt_switches:    130270
nonvoluntary_ctxt_switches: 1686839

What's interesting - there are 2 perf processes (one for DWARF and one for FP) and both have roughly the same amount of used memory. Something causes perf to continue accumulating memory over time, we need to figure it out and either restart perf from time to time, or configure it to prevent this from happening.

Jongy commented 1 year ago

https://github.com/Granulate/gprofiler/pull/613 helps relieving the problem (if perf becomes too memory heavy, we restart it).

The underlying problem of perf growing indefnitely was not resolved yet, hence I keep the ticket open.