Release LLVM memory after binaries have been loaded into kernel

[yonghong-song]

Currently, for a bcc process, after program is loaded into kernel, typically the program will get/set map data and no more compilation is needed. The llvm related resources (bpf program execution engine, sscanf/snprintf execution engine and llvm context) are not freed though. This actually takes quite bit some memory (more than 2MB RSS memory in one of my examples).

If people have many tiny bcc monitoring tools and the sum of llvm related memory consumption could be substantial.

Maybe we can provide a public interface to release internal compilation resources. This way, application writer can release llvm related memory at a pointer which deems safe from operation point of view.

[yonghong-song]

I did some experiments (jemalloc case): . only free engine_ (execution engine compiling the bpf program), I got 1MB memory saving . additionally, if I also free rwengine(sscanf/snprintf) and llvm ctx, I got total 2.5MB saving. The most memory should be from llvm ctx as no usage of sscanf/snprintf in the app so it should not be finalized.

So I kind of still prefer to have an option to free everything. To free rwengine and llvm ctx, we do need user instructions since we have no idea about whether users will use sscanf/snprintf or not.

Do we have use cases to load the same bpf programs again into kernel in the same module context? This seems rare? If users changed the bpf program source code, typically they will create a different BPF module.

[lilydjwg]

Only 2MB RSS? A snoop tool (e.g. biosnoop) takes more than 130MB in my system, which isn't reasonable since there is not much data in use for both the program itself and Python runtime. I assumed a ~30MB size for them instead.

If the compiling takes memory, I hope there is a way to compile the code beforehand.

[yonghong-song]

The newly added free_bcc_memory() api can release compilation memory immediately after the load.

[lilydjwg]

It didn't do much for me.

I inspect the smaps file and it seems that, the heap occupies 67MB, libLLVM-8.so ~31MB and other libclang* occupy several MBs here and there.

[yonghong-song]

In which platform and how do you build bcc with libLLVM-8.so and libclang*.so? Typically I build with static llvm/clang libraries and hence everything is in libbcc.so.

You can take a look at function bcc_free_memory at bcc_elf.c. The only file it attempts to free memory is the /proc/self/ext (the binary itself) and libbcc.so.

It can easily extend to free libLLVM libclang library as well. Maybe you can help contribute. Thanks!

[lilydjwg]

Thank you for the reply. My bcc is dynamically linked to libLLVM-8.so.

I tried to madvise those libraries using ctypes from Python and it seemd to work (RSS significantly dropped), except that it segfaulted on exiting at BPFModule::~BPFModule. I'll look into this further later.

[yonghong-song]

Great!

[springzfx]

Is this still active? The memory is still not much alleviated after bcc_free_memory. Tested on archlinux, dynamic linked to libllvm and libbcc.

[yonghong-song]

The current implementation assumes dynamically linked libbcc.so, but statically linked llvm. So it tried to free insn memory in libbcc.so only. If you also dynamically link llvm as well, the current mechanism may not help.

The commit is here which implemented original bcc_free_memory.

commit 51480d0597cc984e621fe3b38f657a3434dc8712
Author: yonghong-song <ys114321@gmail.com>
Date:   Sun Dec 30 08:57:57 2018 -0800

    implement free_bcc_memory() API (#2097)
...

Maybe you could help improve the implement to free libllvm.so as well for your use case?

[raoufkh]

Hello @yonghong-song

I followed the BCC INSTALL page to install BCC on alpine container, can you provide instructions to statically link llvm?

Regards, Abderaouf

[yonghong-song]

By default, libbcc.so does use static linking for llvm libraries. For example, on my system, I have

$ ldd /usr/lib64/libbcc.so.0.21.0                                                    
        linux-vdso.so.1 (0x00007ffe365cf000)                                                                             
        libelf.so.1 => /lib64/libelf.so.1 (0x00007fd482045000)                                                           
        libdebuginfod.so.1 => /lib64/libdebuginfod.so.1 (0x00007fd481e3f000)                                             
        librt.so.1 => /lib64/librt.so.1 (0x00007fd481c37000)                                                             
        libdl.so.2 => /lib64/libdl.so.2 (0x00007fd481a33000)                                                             
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fd481813000)                                                   
        libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007fd4815e6000)                                                       
        libz.so.1 => /lib64/libz.so.1 (0x00007fd4813cf000)                                                               
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fd48103a000)                                                     
        libm.so.6 => /lib64/libm.so.6 (0x00007fd480cb8000)                                                               
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fd480aa0000)                                                       
        libc.so.6 => /lib64/libc.so.6 (0x00007fd4806db000)                                                               
        /lib64/ld-linux-x86-64.so.2 (0x00007fd4896a1000)                                                                 
        libcurl.so.4 => /lib64/libcurl.so.4 (0x00007fd48044d000)                                                         
        libnghttp2.so.14 => /lib64/libnghttp2.so.14 (0x00007fd480226000)
        libidn2.so.0 => /lib64/libidn2.so.0 (0x00007fd480008000)
        libssh.so.4 => /lib64/libssh.so.4 (0x00007fd47fd99000)
        libpsl.so.5 => /lib64/libpsl.so.5 (0x00007fd47fb88000)
        libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007fd47f8f4000)
        libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007fd47f40e000)
        libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fd47f1b9000)
        libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fd47eed0000)
        libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fd47ecb9000)
        libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fd47eab5000)
        libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007fd47e867000)
        liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007fd47e657000)
        libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007fd47e44a000)
        libunistring.so.2 => /lib64/libunistring.so.2 (0x00007fd47e0c9000)
        libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fd47deb8000)
        libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fd47dcb4000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fd47da9d000)
        libsasl2.so.3 => /lib64/libsasl2.so.3 (0x00007fd47d87f000)
        libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007fd47d65e000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fd47d434000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fd47d20b000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007fd47cf87000)

There are no llvm or clang shared library. If you want to have libbcc.so dynamically linking with clang/llvm library, you need to enable ENABLE_LLVM_SHARED in cmake command line.

Does this answer your question for static linking? Note that we will need to build libbcc.so for python C binding.