[BUG] When a bpf userland program traces itself via the syscall tracepoint, bpftime always crashes

[agentzh]

Consider a bpf userland program traces itself like this:

...
#include <bpf/bpf.h>
#include <bpf/libbpf.h>

int
main(int argc, char *argv[])
{
   ...
    obj = bpf_object__open_file("hello.bpf", NULL);
    ...
    prog = bpf_object__find_program_by_name(obj, "hello");
    ...
    bpf_links[nlinks] = bpf_program__attach_tracepoint(prog, "syscalls", "sys_enter_geteuid");
    ...
    (void) geteuid();
    ...
}

This program is compiled into the executable ./hello.

And in the bpf program:

SEC(".ylang")
int __attribute__((noinline))
hello(void *ctx)
{
    ...
}

And this is compiled into the bpf bytecode object file ./hello.bpf.

Then run it by preloading 2 .so files at the same time:

LIBBPF_USE_ORBPF_KO= LD_PRELOAD=$$HOME/.bpftime/libbpftime-agent.so:$$HOME/.bpftime/libbpftime-syscall-server.so ./hello

It will always crash with the following error:

[2024-01-27 11:26:05.637] [info] [agent.cpp:66] Entering bpftime agent
terminate called after throwing an instance of 'boost::interprocess::interprocess_exception'
  what():  Permission denied

Using gdb to analyze the core dump file:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007fbdbc3fa895 in __GI_abort () at abort.c:79
#2  0x00007fbdbc108941 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /lib64/libstdc++.so.6
#3  0x00007fbdbc11432c in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6
#4  0x00007fbdbc114397 in std::terminate() () from /lib64/libstdc++.so.6
#5  0x00007fbdbc114649 in __cxa_throw () from /lib64/libstdc++.so.6
#6  0x00007fbdbcad69e7 in void boost::interprocess::ipcdetail::managed_open_or_create_impl<boost::interprocess::shared_memory_object, 16ul, true, false>::priv_open_or_create<boost::interprocess::ipcdetail::create_open_func<boost::interprocess::ipcdetail::basic_managed_memory_impl<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index, 16ul> > >(boost::interprocess::ipcdetail::create_enum_t, char const* const&, unsigned long, boost::interprocess::mode_t, void const*, boost::interprocess::permissions const&, boost::interprocess::ipcdetail::create_open_func<boost::interprocess::ipcdetail::basic_managed_memory_impl<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index, 16ul> >) () from /home/agentzh/.bpftime/libbpftime-agent.so
#7  0x00007fbdbcacda7d in bpftime::bpftime_shm::bpftime_shm(char const*, bpftime::shm_open_type) () from /home/agentzh/.bpftime/libbpftime-agent.so
#8  0x00007fbdbcace31c in bpftime::bpftime_shm::bpftime_shm(bpftime::shm_open_type) () from /home/agentzh/.bpftime/libbpftime-agent.so
#9  0x00007fbdbcace3c2 in bpftime_initialize_global_shm () from /home/agentzh/.bpftime/libbpftime-agent.so
#10 0x00007fbdbca8a364 in bpftime_agent_main () from /home/agentzh/.bpftime/libbpftime-agent.so
#11 0x00007fbdbca8a65b in bpftime_hooked_main () from /home/agentzh/.bpftime/libbpftime-agent.so
#12 0x00007fbdbc3fc082 in __libc_start_main (main=0x7fbdbca8a630 <bpftime_hooked_main>, argc=1, argv=0x7fff7c864978, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fff7c864968)

Using root to run the same command yields a different error message but still crashes:

[2024-01-27 11:41:02.972] [info] [agent.cpp:66] Entering bpftime agent
terminate called after throwing an instance of 'boost::interprocess::interprocess_exception'
  what():  No such file or directory
make: *** [Makefile:50: test] Aborted

The core dump backtrace is similar:

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007f7e17c49895 in __GI_abort () at abort.c:79
#2  0x00007f7e17957941 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] () from /lib64/libstdc++.so.6
#3  0x00007f7e1796332c in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6
#4  0x00007f7e17963397 in std::terminate() () from /lib64/libstdc++.so.6
#5  0x00007f7e17963649 in __cxa_throw () from /lib64/libstdc++.so.6
#6  0x00007f7e183259e7 in void boost::interprocess::ipcdetail::managed_open_or_create_impl<boost::interprocess::shared_memory_object, 16ul, true, false>::priv_open_or_create<boost::interprocess::ipcdetail::create_open_func<boost::interprocess::ipcdetail::basic_managed_memory_impl<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index, 16ul> > >(boost::interprocess::ipcdetail::create_enum_t, char const* const&, unsigned long, boost::interprocess::mode_t, void const*, boost::interprocess::permissions const&, boost::interprocess::ipcdetail::create_open_func<boost::interprocess::ipcdetail::basic_managed_memory_impl<char, boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, long, unsigned long, 0ul>, 0ul>, boost::interprocess::iset_index, 16ul> >) () from /home/agentzh/.bpftime/libbpftime-agent.so
#7  0x00007f7e1831ca7d in bpftime::bpftime_shm::bpftime_shm(char const*, bpftime::shm_open_type) () from /home/agentzh/.bpftime/libbpftime-agent.so
#8  0x00007f7e1831d31c in bpftime::bpftime_shm::bpftime_shm(bpftime::shm_open_type) () from /home/agentzh/.bpftime/libbpftime-agent.so
#9  0x00007f7e1831d3c2 in bpftime_initialize_global_shm () from /home/agentzh/.bpftime/libbpftime-agent.so
#10 0x00007f7e182d9364 in bpftime_agent_main () from /home/agentzh/.bpftime/libbpftime-agent.so
#11 0x00007f7e182d965b in bpftime_hooked_main () from /home/agentzh/.bpftime/libbpftime-agent.so
#12 0x00007f7e17c4b082 in __libc_start_main (main=0x7f7e182d9630 <bpftime_hooked_main>, argc=1, argv=0x7ffe7f9a1e98, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffe7f9a1e88)
    at ../csu/libc-start.c:308
#13 0x000000000040174e in _start ()

How to work around this issue? I'm using the latest git master branch HEAD (commit d0b9f6608fc8) on Fedora Linux x86_64. The test suite is almost passing except 2 test failures. And the example/malloc sample in the repo works as expected on my side (without sudo).

[yunwei37]

Related to #194

[Officeyutong]

Investigation:

• syscall server should be put in front of agent
• If we inject both syscall server and inject through LD_PRELOAD, syscall server seems to be loaded twice, and agent won't work
• If we inject agent through remote attach (bpftime attach), it works well