Possible bcc memory leak

[myllynen]

Testing with cat ./test.py ; python3 ./test.py & pid=$! ; sleep 1 ; grep VmRSS /proc/$pid/status ; sleep 60 ; grep VmRSS /proc/$pid/status ; kill $pid ; we see this output:

import gc
from bcc import BPF
while True:
    bpf = BPF(text="BPF_ARRAY(t);")
    bpf.cleanup()
    gc.collect()
VmRSS:     91828 kB
VmRSS:     93388 kB

1) It seems that there is some minor memory leak during bpf.cleanup(), this is irrelevant for simple scripts but for long-running daemons that dynamically attach/detach BPF programs this is more relevant.

2) Overall memory consumption is perhaps a bit high, for example on my laptop only gnome-shell and firefox are currently using more memory (RSS) than the most simple BCC example (pidpersec), processes like X, totem (playing audio), NetworkManager, etc are using less memory.

Thanks.

[yonghong-song]

For 1, I can reproduce a slightely larger gap in the above, but from 56.9M to 59.7M. I will get about 1.5MB gap if I seep 20 seconds instead. Yes, it may be some memory leak somewhere worth investigating.

For 2, most RSS is from llvm insn. The llvm compilation takes humongous number of memory just to bring insn from file to memory. In my case it is roughly 40MB IIRC.

[palmtenor]

What is the baseline RSS (i.e. before first BPF load)? Yes the compilation takes memory, but after the entire things got destructed everything should be gone, at least it's the case in our C++ application. 90+ MB definitely seem high.

[myllynen]

For a pretty much no-op script with Python 3 I see:

# cat foo.py ; python3 ./foo.py & pid=$! ; sleep 2 ; grep VmRSS /proc/$pid/status
import gc
import time

gc.collect()
time.sleep(5)
VmRSS:      8316 kB

For a trivial BPF program we see:

# cat test.py ; python3 ./test.py & pid=$! ; sleep 2 ; for i in 1 2 3 4 ; do grep VmRSS /proc/$pid/status ; sleep 5 ; done
import gc
import time
from bcc import BPF

print("sleep 1")
time.sleep(5)
bpf = BPF(text="BPF_ARRAY(t);")
print("sleep 2")
time.sleep(5)
bpf.cleanup()
print("sleep 3")
time.sleep(5)
gc.collect()
print("sleep 4")
time.sleep(5)
sleep 1
VmRSS:     48456 kB
sleep 2
VmRSS:     94728 kB
sleep 3
VmRSS:     95452 kB
sleep 4
VmRSS:     95452 kB

a) A mere from bcc import BPF increases memory use for a process from 8316 kB -> 48456 kB b) Loading a trivial BPF program increases memory use from 48456 kB -> 94728 kB c) cleanup() increases (?) memory use from 94728 kB -> 95452 kB d) In this case gc.collect() seems to have no effect

Thanks.

[pchaigno]

I'm unable to reproduce with sudo:

$ cat test.py ; sudo python ./test.py & pid=$! ; sleep 2 ; for i in 1 2 3 4 ; do grep VmRSS /proc/$pid/status ; sleep 5 ; done
import gc
import time
from bcc import BPF

print("sleep 1")
time.sleep(5)
bpf = BPF(text="BPF_ARRAY(t);")
print("sleep 2")
time.sleep(5)
bpf.cleanup()
print("sleep 3")
time.sleep(5)
gc.collect()
print("sleep 4")
time.sleep(5)

sleep 1
VmRSS:      4308 kB
sleep 2
VmRSS:      4308 kB
sleep 3
VmRSS:      4308 kB
sleep 4
VmRSS:      4308 kB

But I can if I run as root...

$ sudo su
# python ./test.py & pid=$! ; sleep 2 ; for i in 1 2 3 4 ; do grep VmRSS /proc/$pid/status ; sleep 5 ; done
sleep 1
VmRSS:     23416 kB
sleep 2
VmRSS:     56904 kB
sleep 3
VmRSS:     60784 kB
sleep 4
VmRSS:     60784 kB

@myllynen Same weirdness?

[myllynen]

@pchaigno I think you're grepping sudo not python proc file there.

[thegedge]

I'm likely experiencing the same symptom with gobpf. Application needs about 6M of memory without any BPF probes, but a couple of modules and it jumps to ~70M. I tried Stop()-ing the modules followed by debug.FreeOSMemory(), and the memory didn't seem to disappear. Only after the program terminates does it return to normal levels.

Not sure if this is related to the C bindings in bcc, or gobpf.

[yonghong-song]

The extra memory could be mostly from llvm instructions. bcc implements a API free_bcc_memory which applications can free these memories. Maybe you want to try this?

[thegedge]

bcc implements a API free_bcc_memory which applications can free these memories. Maybe you want to try this?

Looks like gobpf doesn't call this function at all, so I'll give that a try and send a PR to that repo if that's where the memory is. Thanks for the suggestion!