search

eunomia-bpf / bpftime

Userspace eBPF runtime for fast Uprobe & Syscall hook & Extensions with LLVM JIT
https://eunomia.dev/bpftime/
MIT License
687 stars 68 forks source link

[BUG] LLVM JIT cannot be used together with bpftrace #270

Closed yunwei37 closed 2 months ago

yunwei37 commented 2 months ago

Describe the bug

If we compile the bpftime with llvm jit, and try to run bpftrace, it will results in error because there are duplicated llvm library.

NobinPegasus commented 2 months ago

@yunwei37 can you share some more details. I would like to work on this. Like some error message or log. What files have duplicated llvm libraries? Or what commands you use to get the error. SO that I can reproduce it on my local machine.

yunwei37 commented 2 months ago

Could you please try running bpftrace examples with LLVM jit enabled?

yunwei37 commented 2 months ago

The bpftrace commands can be found in example/bpftrace. For how to build bpftime with llvm jit, you can refer to the documents in https://eunomia.dev/bpftime

NobinPegasus commented 2 months ago

@yunwei37 I think bpftime now by default compiles with LLVM JIT enabled? I was looking at these examples: https://github.com/eunomia-bpf/bpftime/tree/master/example/bpftrace

pegasus@pegasus:~/Documents/bpftime$ LD_PRELOAD=./build/runtime/syscall-server/libbpftime-syscall-server.so
pegasus@pegasus:~/Documents/bpftime$ sudo ~/.bpftime/bpftime load bpftrace -e 'uretprobe:/bin/bash:readline { printf("%-6d %s\n", pid, str(retval)); }'
[2024-04-14 16:27:43.219] [error] Library not found: /root/.bpftime/libbpftime-syscall-server.so

I'm getting this error

Officeyutong commented 2 months ago

@yunwei37 I think bpftime now by default compiles with LLVM JIT enabled? I was looking at these examples: https://github.com/eunomia-bpf/bpftime/tree/master/example/bpftrace

pegasus@pegasus:~/Documents/bpftime$ LD_PRELOAD=./build/runtime/syscall-server/libbpftime-syscall-server.so
pegasus@pegasus:~/Documents/bpftime$ sudo ~/.bpftime/bpftime load bpftrace -e 'uretprobe:/bin/bash:readline { printf("%-6d %s\n", pid, str(retval)); }'
[2024-04-14 16:27:43.219] [error] Library not found: /root/.bpftime/libbpftime-syscall-server.so

I'm getting this error

Using sudo means bpftime cli will try to find libraries under /root/.bpftime. Make sure you have installed bpftime for root user

NobinPegasus commented 2 months ago

@Officeyutong I have my path set for sudo.

pegasus@pegasus:~/Documents/bpftime$ LD_PRELOAD=./build/runtime/syscall-server/libbpftime-syscall-server.so
pegasus@pegasus:~/Documents/bpftime$ sudo ~/.bpftime/bpftime load bpftrace -e 'uretprobe:/bin/bash:readline { printf("%-6d %s\n", pid, str(retval)); }'
[2024-04-14 17:39:31.730] [error] Library not found: /root/.bpftime/libbpftime-syscall-server.so
pegasus@pegasus:~/Documents/bpftime$ sudo bpftime
Usage: bpftime [--help] [--version] --install-location VAR [--dry-run] {attach,detach,load,start}

Optional arguments:
  -h, --help              shows help message and exits 
  -v, --version           prints version information and exits 
  -i, --install-location  Installing location of bpftime [default: "/root/.bpftime"]
  -d, --dry-run           Run without commiting any modifications 

Subcommands:
  attach                 Inject bpftime-agent to a certain pid
  detach                 Detach all attached agents
  load                   Start an application with bpftime-server injected
  start                  Start an application with bpftime-agent injected
NobinPegasus commented 2 months ago

I've added /home/pegasus/.bpftime Inside secure_path. image

Officeyutong commented 2 months ago

@Officeyutong I have my path set for sudo.

pegasus@pegasus:~/Documents/bpftime$ LD_PRELOAD=./build/runtime/syscall-server/libbpftime-syscall-server.so
pegasus@pegasus:~/Documents/bpftime$ sudo ~/.bpftime/bpftime load bpftrace -e 'uretprobe:/bin/bash:readline { printf("%-6d %s\n", pid, str(retval)); }'
[2024-04-14 17:39:31.730] [error] Library not found: /root/.bpftime/libbpftime-syscall-server.so
pegasus@pegasus:~/Documents/bpftime$ sudo bpftime
Usage: bpftime [--help] [--version] --install-location VAR [--dry-run] {attach,detach,load,start}

Optional arguments:
  -h, --help              shows help message and exits 
  -v, --version           prints version information and exits 
  -i, --install-location  Installing location of bpftime [default: "/root/.bpftime"]
  -d, --dry-run           Run without commiting any modifications 

Subcommands:
  attach                 Inject bpftime-agent to a certain pid
  detach                 Detach all attached agents
  load                   Start an application with bpftime-server injected
  start                  Start an application with bpftime-agent injected

Okay, but bpftime cli currently uses $HOME/.bpftime for the directory to store libraries. If you run bpftime with sudo, $HOME will be set to /root for root user. You may try sudo -eto keep environment variables, or you may install bpftime for root user by using sudo from the beginning of installing. You may also use argument -i to force a directory for libraries

NobinPegasus commented 2 months ago

@Officeyutong 

pegasus@pegasus:~$ sudo -E ~/.bpftime/bpftime load bpftrace -e 'uretprobe:/bin/bash:readline { printf("%-6d %s\n", pid, str(retval)); }'
[2024-04-15 00:58:27.894] [info] [syscall_context.hpp:86] manager constructed
[2024-04-15 00:58:30.447] [info] [syscall_server_utils.cpp:24] Initialize syscall server
[2024-04-15 00:58:30][info][9643] Global shm constructed. shm_open_type 0 for bpftime_maps_shm
[2024-04-15 00:58:30][info][9643] Global shm initialized
[2024-04-15 00:58:30][info][9643] Enabling helper groups ufunc, kernel, shm_map by default
[2024-04-15 00:58:30][info][9643] bpftime-syscall-server started
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
pegasus@pegasus:~$ sudo -E SPDLOG_LEVEL=error ~/.bpftime/bpftime load bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'
[sudo] password for pegasus: 
[2024-04-15 00:59:30.259] [info] [syscall_context.hpp:86] manager constructed
[2024-04-15 00:59:32.851] [info] [syscall_server_utils.cpp:24] Initialize syscall server
libclang: crash detected during parsing: {
  'source_filename' : 'definitions.h'
  'command_line_args' : ['clang', '-isystem', '/bpftrace/include', '-isystem', '/usr/local/include', '-isystem', '/usr/include', '-isystem', '/usr/include/x86_64-linux-gnu', '-nostdinc', '-isystem', '/virtual/lib/clang/include', '-I/lib/modules/6.4.0-060400-generic/build/arch/x86/include', '-I/lib/modules/6.4.0-060400-generic/build/arch/x86/include/generated', '-I/lib/modules/6.4.0-060400-generic/build/include', '-I/lib/modules/6.4.0-060400-generic/build/include', '-I/lib/modules/6.4.0-060400-generic/build/arch/x86/include/uapi', '-I/lib/modules/6.4.0-060400-generic/build/arch/x86/include/generated/uapi', '-I/lib/modules/6.4.0-060400-generic/build/include/uapi', '-I/lib/modules/6.4.0-060400-generic/build/include/generated/uapi', '-include', '/lib/modules/6.4.0-060400-generic/build/include/linux/kconfig.h', '-D__KERNEL__', '-D__BPF_TRACING__', '-D__HAVE_BUILTIN_BSWAP16__', '-D__HAVE_BUILTIN_BSWAP32__', '-D__HAVE_BUILTIN_BSWAP64__', '-DKBUILD_MODNAME="bpftrace"', '-include', 'clang_workarounds.h'],
  'unsaved_files' : [('definitions.h', '...', 380), ('/bpftrace/include/__stddef_max_align_t.h', '...', 1771), ('/bpftrace/include/float.h', '...', 5192), ('/bpftrace/include/limits.h', '...', 3735), ('/bpftrace/include/stdarg.h', '...', 2025), ('/bpftrace/include/stdbool.h', '...', 1010), ('/bpftrace/include/stddef.h', '...', 4499), ('/bpftrace/include/stdint.h', '...', 23390), ('/bpftrace/include/clang_workarounds.h', '...', 921), ('/bpftrace/include/__btf_generated_header.h', '...', 0)],
  'options' : 1,
}
INFO [10547]: Global shm destructed

This is the errors that I got!

Officeyutong commented 2 months ago

@Officeyutong 


pegasus@pegasus:~$ sudo -E ~/.bpftime/bpftime load bpftrace -e 'uretprobe:/bin/bash:readline { printf("%-6d %s\n", pid, str(retval)); }'

[2024-04-15 00:58:27.894] [info] [syscall_context.hpp:86] manager constructed

[2024-04-15 00:58:30.447] [info] [syscall_server_utils.cpp:24] Initialize syscall server

[2024-04-15 00:58:30][info][9643] Global shm constructed. shm_open_type 0 for bpftime_maps_shm

[2024-04-15 00:58:30][info][9643] Global shm initialized

[2024-04-15 00:58:30][info][9643] Enabling helper groups ufunc, kernel, shm_map by default

[2024-04-15 00:58:30][info][9643] bpftime-syscall-server started

terminate called after throwing an instance of 'std::bad_alloc'

  what():  std::bad_alloc


pegasus@pegasus:~$ sudo -E SPDLOG_LEVEL=error ~/.bpftime/bpftime load bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

[sudo] password for pegasus: 

[2024-04-15 00:59:30.259] [info] [syscall_context.hpp:86] manager constructed

[2024-04-15 00:59:32.851] [info] [syscall_server_utils.cpp:24] Initialize syscall server

libclang: crash detected during parsing: {

  'source_filename' : 'definitions.h'

  'command_line_args' : ['clang', '-isystem', '/bpftrace/include', '-isystem', '/usr/local/include', '-isystem', '/usr/include', '-isystem', '/usr/include/x86_64-linux-gnu', '-nostdinc', '-isystem', '/virtual/lib/clang/include', '-I/lib/modules/6.4.0-060400-generic/build/arch/x86/include', '-I/lib/modules/6.4.0-060400-generic/build/arch/x86/include/generated', '-I/lib/modules/6.4.0-060400-generic/build/include', '-I/lib/modules/6.4.0-060400-generic/build/include', '-I/lib/modules/6.4.0-060400-generic/build/arch/x86/include/uapi', '-I/lib/modules/6.4.0-060400-generic/build/arch/x86/include/generated/uapi', '-I/lib/modules/6.4.0-060400-generic/build/include/uapi', '-I/lib/modules/6.4.0-060400-generic/build/include/generated/uapi', '-include', '/lib/modules/6.4.0-060400-generic/build/include/linux/kconfig.h', '-D__KERNEL__', '-D__BPF_TRACING__', '-D__HAVE_BUILTIN_BSWAP16__', '-D__HAVE_BUILTIN_BSWAP32__', '-D__HAVE_BUILTIN_BSWAP64__', '-DKBUILD_MODNAME="bpftrace"', '-include', 'clang_workarounds.h'],

  'unsaved_files' : [('definitions.h', '...', 380), ('/bpftrace/include/__stddef_max_align_t.h', '...', 1771), ('/bpftrace/include/float.h', '...', 5192), ('/bpftrace/include/limits.h', '...', 3735), ('/bpftrace/include/stdarg.h', '...', 2025), ('/bpftrace/include/stdbool.h', '...', 1010), ('/bpftrace/include/stddef.h', '...', 4499), ('/bpftrace/include/stdint.h', '...', 23390), ('/bpftrace/include/clang_workarounds.h', '...', 921), ('/bpftrace/include/__btf_generated_header.h', '...', 0)],

  'options' : 1,

}

INFO [10547]: Global shm destructed

This is the errors that I got!

The first error is caused by the predefined shared memory size being too small. I'll add an environment variable to define that size at runtime. The second one seems to be related to your bpftrace, and not related to bpftime

NobinPegasus commented 2 months ago

@Officeyutong I've used 10GB as shared memory by editing 

    } else if (type == shm_open_type::SHM_CREATE_OR_OPEN) {
        SPDLOG_DEBUG("start: bpftime_shm for create or open setup");
        segment = boost::interprocess::managed_shared_memory(
            boost::interprocess::open_or_create,
            // Allocate 20M bytes of memory by default
            shm_name, 10000 << 20);

inside bpftime_shm_internal.hpp. Still getting the same error

pegasus@pegasus:~$ sudo -E ~/.bpftime/bpftime load bpftrace -e 'uretprobe:/bin/bash:readline { printf("%-6d %s\n", pid, str(retval)); }'
[2024-04-18 04:22:31.577] [info] [syscall_context.hpp:86] manager constructed
[2024-04-18 04:22:35.082] [info] [syscall_server_utils.cpp:24] Initialize syscall server
[2024-04-18 04:22:35][info][20624] Global shm constructed. shm_open_type 0 for bpftime_maps_shm
[2024-04-18 04:22:35][info][20624] Global shm initialized
[2024-04-18 04:22:35][info][20624] Enabling helper groups ufunc, kernel, shm_map by default
[2024-04-18 04:22:35][info][20624] bpftime-syscall-server started
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Officeyutong commented 2 months ago

@Officeyutong I've used 10GB as shared memory by editing

  } else if (type == shm_open_type::SHM_CREATE_OR_OPEN) {
      SPDLOG_DEBUG("start: bpftime_shm for create or open setup");
      segment = boost::interprocess::managed_shared_memory(
          boost::interprocess::open_or_create,
          // Allocate 20M bytes of memory by default
          shm_name, 10000 << 20);

inside bpftime_shm_internal.hpp. Still getting the same error

pegasus@pegasus:~$ sudo -E ~/.bpftime/bpftime load bpftrace -e 'uretprobe:/bin/bash:readline { printf("%-6d %s\n", pid, str(retval)); }'
[2024-04-18 04:22:31.577] [info] [syscall_context.hpp:86] manager constructed
[2024-04-18 04:22:35.082] [info] [syscall_server_utils.cpp:24] Initialize syscall server
[2024-04-18 04:22:35][info][20624] Global shm constructed. shm_open_type 0 for bpftime_maps_shm
[2024-04-18 04:22:35][info][20624] Global shm initialized
[2024-04-18 04:22:35][info][20624] Enabling helper groups ufunc, kernel, shm_map by default
[2024-04-18 04:22:35][info][20624] bpftime-syscall-server started
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

I can't reproduce your issue, could please debug with gdb and check where the exception throws? You may use LD_PRELOAD to inject bpftime-syscall-server into instead of using bpftime, which might be more convenient for debugging, since it avoids subprocesses

Officeyutong commented 2 months ago

One of repdocution:

root@mnfe-pve:~/bpftime# bpftime load ../bpftrace/build/src/bpftrace 
[2024-04-21 23:50:58.748] [info] [syscall_context.hpp:86] manager constructed
: CommandLine Error: Option 'enable-partial-inlining' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options

bpftrace uses LLVM 14, while bpftime uses LLVM 15

This might be happened when some parts of LLVM being initialized twice