how to use memleak tool to detect memory leak of process which alloctor is jemalloc or tcmalloc

[judeng]

Hi, everyone, I'm a beginner of bcc. bcc is powerful, and I'm trying to use it to detect redis's memory leak in the production environment. But redis use jemalloc as default alloctor, memleak always output nothing. I also tried to specify the allocator for jemalloc with the -O argument, still failed.

$ sudo memleak-bpfcc -p 34610 -O ./libjemalloc.a
Attaching to pid 34610, Ctrl+C to quit.
Traceback (most recent call last):
  File "/usr/sbin/memleak-bpfcc", line 449, in <module>
    attach_probes("malloc")
  File "/usr/sbin/memleak-bpfcc", line 437, in attach_probes
    bpf.attach_uprobe(name=obj, sym=sym,
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 1144, in attach_uprobe
    (path, addr) = BPF._check_path_symbol(name, sym, addr, pid, sym_off)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 781, in _check_path_symbol
    raise Exception("could not determine address of symbol %s" % symname)
Exception: could not determine address of symbol b'malloc'

I tried to make simple changes to the source code, add je_ prefix for the probe names, also failed :(

        attach_probes("je_malloc")                                                                                                                                          
        attach_probes("je_calloc")
        attach_probes("je_realloc")
        attach_probes("je_posix_memalign")
        attach_probes("valloc", can_fail=True) # failed on Android, is deprecated in libc.so from bionic directory
        attach_probes("je_memalign")
        attach_probes("pvalloc", can_fail=True) # failed on Android, is deprecated in libc.so from bionic directory
        attach_probes("aligned_alloc", can_fail=True)  # added in C11
        bpf.attach_uprobe(name=obj, sym="je_free", fn_name="free_enter",
                                  pid=pid)

so I want to known what is the correct usage of memleak for jemalloc?

I don't know if this is the right place to ask a question, please feel free to close this issue, thank you

[chenhengqi]

I haven't played with jemalloc, but for tcmalloc, the memleak tool won't work even if you specify--obj=/path/to/libtcmalloc.so. That's because the C APIs is just a thin wrapper around the C++ implementation.

For jemalloc, there may be similar issues.

[judeng]

That's because the C APIs is just a thin wrapper around the C++ implementation.

@chenhengqi thanks, could we directly hook these real implementations? such as jemalloc, because the prefix is "je_", so the real malloc function is "je_malloc", I don't understand why the attach je_malloc function still fails

$ nm -a libjemalloc.a |grep je_malloc
00000000000085f0 T je_malloc

[chenhengqi]

Please paste the error messages here.

[judeng]

@chenhengqi sorry for the late, here is complete logs, thank you

sudo ./memleak-bpfcc -p 1924
In file included from <built-in>:2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
In file included from include/uapi/linux/types.h:14:
In file included from include/uapi/linux/posix_types.h:5:
In file included from include/linux/stddef.h:5:
In file included from include/uapi/linux/stddef.h:5:
In file included from include/linux/compiler_types.h:122:
include/linux/compiler-clang.h:41:9: warning: '__HAVE_BUILTIN_BSWAP32__' macro redefined [-Wmacro-redefined]
#define __HAVE_BUILTIN_BSWAP32__
        ^
<command line>:4:9: note: previous definition is here
#define __HAVE_BUILTIN_BSWAP32__ 1
        ^
In file included from <built-in>:2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
In file included from include/uapi/linux/types.h:14:
In file included from include/uapi/linux/posix_types.h:5:
In file included from include/linux/stddef.h:5:
In file included from include/uapi/linux/stddef.h:5:
In file included from include/linux/compiler_types.h:122:
include/linux/compiler-clang.h:42:9: warning: '__HAVE_BUILTIN_BSWAP64__' macro redefined [-Wmacro-redefined]
#define __HAVE_BUILTIN_BSWAP64__
        ^
<command line>:5:9: note: previous definition is here
#define __HAVE_BUILTIN_BSWAP64__ 1
        ^
In file included from <built-in>:2:
In file included from /virtual/include/bcc/bpf.h:12:
In file included from include/linux/types.h:6:
In file included from include/uapi/linux/types.h:14:
In file included from include/uapi/linux/posix_types.h:5:
In file included from include/linux/stddef.h:5:
In file included from include/uapi/linux/stddef.h:5:
In file included from include/linux/compiler_types.h:122:
include/linux/compiler-clang.h:43:9: warning: '__HAVE_BUILTIN_BSWAP16__' macro redefined [-Wmacro-redefined]
#define __HAVE_BUILTIN_BSWAP16__
        ^
<command line>:3:9: note: previous definition is here
#define __HAVE_BUILTIN_BSWAP16__ 1
        ^
3 warnings generated.
Attaching to pid 1924, Ctrl+C to quit.
Traceback (most recent call last):
  File "/home/jude/./memleak-bpfcc", line 449, in <module>
    attach_probes("je_malloc")
  File "/home/jude/./memleak-bpfcc", line 437, in attach_probes
    bpf.attach_uprobe(name=obj, sym=sym,
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 1144, in attach_uprobe
    (path, addr) = BPF._check_path_symbol(name, sym, addr, pid, sym_off)
  File "/usr/lib/python3/dist-packages/bcc/__init__.py", line 781, in _check_path_symbol
    raise Exception("could not determine address of symbol %s" % symname)
Exception: could not determine address of symbol b'je_malloc'

[judeng]

@chenhengqi hi, yesterday I read the developer tutorial and make some small changes in the memleak, now it works when compiling witch -fno-omit-frame-point.

sudo python3 ./redis_memleak.py 10 -p 10367 -O '/proc/10367/exe'
Attaching to pid 10367, Ctrl+C to quit.
[17:13:11] Top 10 stacks with outstanding allocations:
    94 bytes in 94 allocations from stack
        0x0000558b2278d39f  zmalloc+0x2f [redis-server]
        0x0000558b2277e6ee  serverCron+0x2e [redis-server]
        0x0000558b2276ce1b  processTimeEvents+0x5b [redis-server]
        0x0000558b2276de60  aeMain+0x1d0 [redis-server]
        0x0000558b227670b7  main+0x4a7 [redis-server]
        0x00007f8054a29d90  libc_start_call_main+0x80 [libc.so.6]
[17:13:21] Top 10 stacks with outstanding allocations:
    194 bytes in 194 allocations from stack
        0x0000558b2278d39f  zmalloc+0x2f [redis-server]
        0x0000558b2277e6ee  serverCron+0x2e [redis-server]
        0x0000558b2276ce1b  processTimeEvents+0x5b [redis-server]
        0x0000558b2276de60  aeMain+0x1d0 [redis-server]
        0x0000558b227670b7  main+0x4a7 [redis-server]
        0x00007f8054a29d90  libc_start_call_main+0x80 [libc.so.6]

memleak could easily support redis's memory leak because redis static link the jemalloc, and always use the prefix "je_", do you feel the need to make a pr for adaptation?

[judeng]

Also, can we print a part of the memory leak, say 16byte? When the compiler optimizes to remove the frame pointer, current memleak is still hard to discover intuitively the location of a memory leak

[chenhengqi]

PR is always welcome.