search

Sysinternals / ProcDump-for-Linux

A Linux version of the ProcDump Sysinternals tool
MIT License
2.91k stars 302 forks source link

3.2.0: build fails with `libbpf: failed to find BTF info for object procdump_ebpf.o` #241

Open kloczek opened 3 months ago

kloczek commented 3 months ago

cmake settings

-- Cache values
CMAKE_BUILD_TYPE:STRING=RelWithDebInfo
CMAKE_INSTALL_PREFIX:PATH=/usr
PROCDUMP_DISABLE_SYSTEM_LIBBPF:BOOL=OFF

cmake params and output:

+ /usr/bin/cmake -B x86_64-redhat-linux-gnu -D BUILD_SHARED_LIBS=ON -D CMAKE_AR=/usr/bin/gcc-ar -D CMAKE_BUILD_TYPE=RelWithDebInfo -D CMAKE_C_FLAGS_RELEASE=-DNDEBUG -D CMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -D CMAKE_Fortran_FLAGS_RELEASE=-DNDEBUG -D CMAKE_INSTALL_PREFIX=/usr -D CMAKE_NM=/usr/bin/gcc-nm -D CMAKE_RANLIB=/usr/bin/gcc-ranlib -D CMAKE_VERBOSE_MAKEFILE=ON -D INCLUDE_INSTALL_DIR=/usr/include -D LIB_INSTALL_DIR=/usr/lib64 -D LIB_SUFFIX=64 -D SHARE_INSTALL_PREFIX=/usr/share -D SYSCONF_INSTALL_DIR=/etc -S .
-- The C compiler identification is GNU 14.0.1
-- The CXX compiler identification is GNU 14.0.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Found ZLIB: /usr/lib64/libz.so (found version "1.3.0")
-- Found Libelf: /usr/lib64/libelf.so (found version "0.191")
-- Found Bpf:  (found version "1.4.0") found components: libbpf
-- Found Bpf:  (found version "1.4.0") found components: bpftool
-- Configuring done (1.5s)
-- Generating done (0.0s)
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_Fortran_FLAGS_RELEASE
    INCLUDE_INSTALL_DIR
    LIB_INSTALL_DIR
    LIB_SUFFIX
    SHARE_INSTALL_PREFIX
    SYSCONF_INSTALL_DIR

And build fails with:

[ 24%] Built target ProcDumpProfiler
make  -f CMakeFiles/procdump_ebpf.dir/build.make CMakeFiles/procdump_ebpf.dir/depend
make[2]: Entering directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
cd /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0 /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0 /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu/CMakeFiles/procdump_ebpf.dir/DependInfo.cmake "--color="
make[2]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
make  -f CMakeFiles/procdump_ebpf.dir/build.make CMakeFiles/procdump_ebpf.dir/build
make[2]: Entering directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
[ 27%] Generating procdump.ebpf.o
/usr/sbin/bpftool gen object procdump.ebpf.o procdump_ebpf.o
libbpf: failed to find BTF info for object 'procdump_ebpf.o'
Error: failed to link 'procdump_ebpf.o': Invalid argument (22)
make[2]: *** [CMakeFiles/procdump_ebpf.dir/build.make:78: procdump.ebpf.o] Error 234
make[2]: *** Deleting file 'procdump.ebpf.o'
make[2]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
make[1]: *** [CMakeFiles/Makefile2:308: CMakeFiles/procdump_ebpf.dir/all] Error 2
make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu'
make: *** [Makefile:94: all] Error 2
MarioHewardt commented 3 months ago

Hi - Thanks for reporting this issue. What distribution is this and does it have BTF enabled (check - /sys/kernel/btf/vmlinux)?

kloczek commented 3 months ago

I'm using my own distribution but on build system is Fedora kernel.

[tkloczko@pers-jacek SPECS]$ ls -la /sys/kernel/btf/vmlinux
-r--r--r-- 1 root root 5923179 Apr 19 15:20 /sys/kernel/btf/vmlinux

Does it mean that what bpftool is doing depends on system running kernel? 🤔

MarioHewardt commented 3 months ago

The -restrack switch (memory tracking) is built using eBPF which utilizes BTF. Having said that, your system seems to be BTF enabled so there is something else going on. We build and produce packages for RHEL so there must be something specific to your distribution.

Try adding --debug to the bpf build steps in the cmake file. Towards the bottom you should see:

bpftool gen object procdump.ebpf.o procdump_ebpf.o

Change it to:

bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o

This may shed some more light on the problem.

kloczek commented 3 months ago 
[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o; echo

libbpf: linker: adding object file 'procdump_ebpf.o'...
libbpf: failed to find BTF info for object 'procdump_ebpf.o'
Error: failed to link 'procdump_ebpf.o': Invalid argument (22)

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ ls -la procdump_ebpf.o
-rw-r--r-- 1 tkloczko tkloczko 27104 Apr 19 15:33 procdump_ebpf.o
kloczek commented 3 months ago

I've been trying to test that using strace ..

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ strace -fe trace=file bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o
execve("/usr/sbin/bpftool", ["bpftool", "--debug", "gen", "object", "procdump.ebpf.o", "procdump_ebpf.o"], 0x7fff868ff1f8 /* 30 vars */) = 0
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libelf.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib64/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "procdump.ebpf.o", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0644) = 3
libbpf: linker: adding object file 'procdump_ebpf.o'...
openat(AT_FDCWD, "procdump_ebpf.o", O_RDONLY|O_CLOEXEC) = 4
libbpf: failed to find BTF info for object 'procdump_ebpf.o'
Error: failed to link 'procdump_ebpf.o': Invalid argument (22)
+++ exited with 234 +++

So noting is trying to access to sysfs so kernel version and availability of the /sys/kernel/btf/vmlinux should not be relevant.

kloczek commented 3 months ago

I've tested as well use bpdtools instead from package generated by me use package fro Fedora rawhide and result is the same

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ rpm -qf /usr/sbin/bpftool; echo
bpftool-6.9.0-0.rc4.20240416git96fca68c4fbf7.38.fc41.x86_64

[tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o
libbpf: linker: adding object file 'procdump_ebpf.o'...
libbpf: failed to find BTF info for object 'procdump_ebpf.o'
Error: failed to link 'procdump_ebpf.o': Invalid argument (22)
kloczek commented 3 months ago

Here is full strace output

```console [tkloczko@pers-jacek x86_64-redhat-linux-gnu]$ strace -f bpftool --debug gen object procdump.ebpf.o procdump_ebpf.o execve("/usr/sbin/bpftool", ["bpftool", "--debug", "gen", "object", "procdump.ebpf.o", "procdump_ebpf.o"], 0x7fffdb955750 /* 30 vars */) = 0 brk(NULL) = 0x560ae84cb000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c3dab9000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=30649, ...}) = 0 mmap(NULL, 30649, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3c3dab1000 close(3) = 0 openat(AT_FDCWD, "/lib64/libelf.so.1", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=111824, ...}) = 0 mmap(NULL, 110608, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3da95000 mmap(0x7f3c3da98000, 77824, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f3c3da98000 mmap(0x7f3c3daab000, 16384, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f3c3daab000 mmap(0x7f3c3daaf000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7f3c3daaf000 mmap(0x7f3c3dab0000, 16, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3c3dab0000 close(3) = 0 openat(AT_FDCWD, "/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=141048, ...}) = 0 mmap(NULL, 139416, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3da72000 mmap(0x7f3c3da74000, 94208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f3c3da74000 mmap(0x7f3c3da8b000, 32768, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19000) = 0x7f3c3da8b000 mmap(0x7f3c3da93000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x20000) = 0x7f3c3da93000 close(3) = 0 openat(AT_FDCWD, "/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0,q\0\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=44920, ...}) = 0 mmap(NULL, 45136, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3da66000 mmap(0x7f3c3da69000, 20480, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f3c3da69000 mmap(0x7f3c3da6e000, 8192, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x8000) = 0x7f3c3da6e000 mmap(0x7f3c3da70000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x7f3c3da70000 close(3) = 0 openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\243\2\0\0\0\0\0"..., 832) = 832 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 fstat(3, {st_mode=S_IFREG|0755, st_size=2433096, ...}) = 0 pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784 mmap(NULL, 2022296, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3d878000 mmap(0x7f3c3d8a0000, 1482752, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f3c3d8a0000 mmap(0x7f3c3da0a000, 319488, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x192000) = 0x7f3c3da0a000 mmap(0x7f3c3da58000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e0000) = 0x7f3c3da58000 mmap(0x7f3c3da5e000, 31640, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3c3da5e000 close(3) = 0 openat(AT_FDCWD, "/lib64/libzstd.so.1", O_RDONLY|O_CLOEXEC) = 3 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=786912, ...}) = 0 mmap(NULL, 782384, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3c3d7b8000 mmap(0x7f3c3d7bc000, 708608, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f3c3d7bc000 mmap(0x7f3c3d869000, 53248, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb1000) = 0x7f3c3d869000 mmap(0x7f3c3d876000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xbd000) = 0x7f3c3d876000 mmap(0x7f3c3d877000, 48, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3c3d877000 close(3) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c3d7b6000 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3c3d7b4000 arch_prctl(ARCH_SET_FS, 0x7f3c3d7b7640) = 0 set_tid_address(0x7f3c3d7b7910) = 1146977 set_robust_list(0x7f3c3d7b7920, 24) = 0 rseq(0x7f3c3d7b7f60, 0x20, 0, 0x53053053) = 0 mprotect(0x7f3c3da58000, 16384, PROT_READ) = 0 mprotect(0x7f3c3d876000, 4096, PROT_READ) = 0 mprotect(0x7f3c3da70000, 4096, PROT_READ) = 0 mprotect(0x7f3c3da93000, 4096, PROT_READ) = 0 mprotect(0x7f3c3daaf000, 4096, PROT_READ) = 0 mprotect(0x560ae6e3b000, 765952, PROT_READ) = 0 mprotect(0x7f3c3daed000, 8192, PROT_READ) = 0 prlimit64(0, RLIMIT_STACK, NULL, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0 munmap(0x7f3c3dab1000, 30649) = 0 prctl(PR_CAPBSET_READ, CAP_MAC_OVERRIDE) = 0 prctl(PR_CAPBSET_READ, 0x30 /* CAP_??? */) = -1 EINVAL (Invalid argument) prctl(PR_CAPBSET_READ, CAP_CHECKPOINT_RESTORE) = 1 prctl(PR_CAPBSET_READ, 0x2c /* CAP_??? */) = -1 EINVAL (Invalid argument) prctl(PR_CAPBSET_READ, 0x2a /* CAP_??? */) = -1 EINVAL (Invalid argument) prctl(PR_CAPBSET_READ, 0x29 /* CAP_??? */) = -1 EINVAL (Invalid argument) getrandom("\xb7\xbc\x71\xd7\x8a\xa7\x44\x63", 8, GRND_NONBLOCK) = 8 brk(NULL) = 0x560ae84cb000 brk(0x560ae84ec000) = 0x560ae84ec000 openat(AT_FDCWD, "procdump.ebpf.o", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0644) = 3 fcntl(3, F_GETFD) = 0x1 (flags FD_CLOEXEC) write(2, "libbpf: linker: adding object fi"..., 56libbpf: linker: adding object file 'procdump_ebpf.o'... ) = 56 openat(AT_FDCWD, "procdump_ebpf.o", O_RDONLY|O_CLOEXEC) = 4 fcntl(4, F_GETFD) = 0x1 (flags FD_CLOEXEC) fstat(4, {st_mode=S_IFREG|0644, st_size=27104, ...}) = 0 pread64(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\1\0\367\0\1\0\0\0\0\0\0\0\0\0\0\0"..., 64, 0) = 64 pread64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2432, 24672) = 2432 pread64(4, "\0uprobe_reallocarray\0uretprobe_r"..., 2579, 22088) = 2579 pread64(4, "{\32\320\377\0\0\0\0y\241\320\377\0\0\0\0{\32\360\377\0\0\0\0\30\1\0\0\0\0\0\0"..., 2712, 64) = 2712 pread64(4, "\30\0\0\0\0\0\0\0\1\0\0\0\332\0\0\0`\0\0\0\0\0\0\0\1\0\0\0\333\0\0\0"..., 144, 20680) = 144 pread64(4, "{\32\250\377\0\0\0\0y\241\250\377\0\0\0\0y\22h\0\0\0\0\0{*\330\377\0\0\0\0"..., 1024, 2776) = 1024 pread64(4, "\200\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\270\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 20824) = 128 pread64(4, "{\32\310\377\0\0\0\0y\241\310\377\0\0\0\0y\21P\0\0\0\0\0{\32\360\377\0\0\0\0"..., 608, 3800) = 608 pread64(4, "p\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\330\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 20952) = 64 pread64(4, "{\32\270\377\0\0\0\0y\241\270\377\0\0\0\0y\21p\0\0\0\0\0{\32\340\377\0\0\0\0"..., 728, 4408) = 728 pread64(4, "p\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\330\0\0\0\0\0\0\0\1\0\0\0\343\0\0\0"..., 64, 21016) = 64 pread64(4, "{\32\310\377\0\0\0\0y\241\310\377\0\0\0\0y\21p\0\0\0\0\0{\32\360\377\0\0\0\0"..., 608, 5136) = 608 pread64(4, "p\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\330\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21080) = 64 pread64(4, "{\32\230\377\0\0\0\0y\242\230\377\0\0\0\0y!p\0\0\0\0\0{*\250\377\0\0\0\0"..., 1056, 5744) = 1056 pread64(4, "\240\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\330\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 21144) = 128 pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 6800) = 632 pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21272) = 64 pread64(4, "{\32\250\377\0\0\0\0y\242\250\377\0\0\0\0y!P\0\0\0\0\0{*\270\377\0\0\0\0"..., 752, 7432) = 752 pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\343\0\0\0"..., 64, 21336) = 64 pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 8184) = 632 pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21400) = 64 pread64(4, "{\32\220\377\0\0\0\0y\243\220\377\0\0\0\0y2p\0\0\0\0\0y1h\0\0\0\0\0"..., 1072, 8816) = 1072 pread64(4, "\260\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\350\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 21464) = 128 pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 9888) = 632 pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21592) = 64 pread64(4, "{\32\220\377\0\0\0\0y\243\220\377\0\0\0\0y2p\0\0\0\0\0y1h\0\0\0\0\0"..., 1072, 10520) = 1072 pread64(4, "\260\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\350\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 21656) = 128 pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 11592) = 632 pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21784) = 64 pread64(4, "{\32\210\377\0\0\0\0y\244\210\377\0\0\0\0yCp\0\0\0\0\0yBh\0\0\0\0\0"..., 1088, 12224) = 1088 pread64(4, "\300\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\370\0\0\0\0\0\0\0\1\0\0\0\341\0\0\0"..., 128, 21848) = 128 pread64(4, "{\32\270\377\0\0\0\0y\242\270\377\0\0\0\0y!P\0\0\0\0\0{*\310\377\0\0\0\0"..., 632, 13312) = 632 pread64(4, "\210\0\0\0\0\0\0\0\1\0\0\0\340\0\0\0\360\0\0\0\0\0\0\0\1\0\0\0\344\0\0\0"..., 64, 21976) = 64 pread64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 144, 13944) = 144 pread64(4, "Dual BSD/GPL\0", 13, 14088) = 13 pread64(4, "[ResourceAllocHelper] Failed: Ge"..., 718, 14104) = 718 pread64(4, "\337\1\345\1\347\1\350\1\351\1\352\1\353\1\354\1\355\1\356\1\357\1\360\1\361\1\362\1\334\1\363\1"..., 48, 22040) = 48 pread64(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0K\3\0\0\4\0\361\377"..., 5856, 14824) = 5856 brk(0x560ae850e000) = 0x560ae850e000 brk(0x560ae850d000) = 0x560ae850d000 write(2, "libbpf: failed to find BTF info "..., 61libbpf: failed to find BTF info for object 'procdump_ebpf.o' ) = 61 close(4) = 0 write(2, "Error: ", 7Error: ) = 7 write(2, "failed to link 'procdump_ebpf.o'"..., 55failed to link 'procdump_ebpf.o': Invalid argument (22)) = 55 write(2, "\n", 1 ) = 1 close(3) = 0 brk(0x560ae8509000) = 0x560ae8509000 exit_group(-22) = ? +++ exited with 234 +++ ```

after delete procdump_ebpf.o':

```console [ 27%] Building EBPF object procdump_ebpf.o clang -nostdinc -isystem `gcc -print-file-name=include` -I /usr/include -I /usr/include/x86_64-linux-gnu -I /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0 -I /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu -I /usr/include -D __KERNEL__ -D __BPF_TRACING__ -D __TARGET_ARCH_x86 -D __linux__ -target bpf -fno-stack-protector -c /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c -o procdump_ebpf.o /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:90:20: warning: left operand of comma operator has no effect [-Wunused-value] 90 | BPF_PRINTK("[SendEvent] Failed: Getting event (allocation address: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:90:103: warning: left operand of comma operator has no effect [-Wunused-value] 90 | BPF_PRINTK("[SendEvent] Failed: Getting event (allocation address: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:90:110: warning: expression result unused [-Wunused-value] 90 | BPF_PRINTK("[SendEvent] Failed: Getting event (allocation address: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:104:20: warning: left operand of comma operator has no effect [-Wunused-value] 104 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:104:120: warning: left operand of comma operator has no effect [-Wunused-value] 104 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ~~~~~ ^~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:104:134: warning: left operand of comma operator has no effect [-Wunused-value] 104 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:104:141: warning: expression result unused [-Wunused-value] 104 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:110:20: warning: left operand of comma operator has no effect [-Wunused-value] 110 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:110:120: warning: left operand of comma operator has no effect [-Wunused-value] 110 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ~~~~~ ^~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:110:134: warning: left operand of comma operator has no effect [-Wunused-value] 110 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:110:141: warning: expression result unused [-Wunused-value] 110 | BPF_PRINTK("[SendEvent] Failed: Getting event (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:114:16: warning: left operand of comma operator has no effect [-Wunused-value] 114 | BPF_PRINTK("[SendEvent] Success: (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:114:103: warning: left operand of comma operator has no effect [-Wunused-value] 114 | BPF_PRINTK("[SendEvent] Success: (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ~~~~~ ^~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:114:117: warning: left operand of comma operator has no effect [-Wunused-value] 114 | BPF_PRINTK("[SendEvent] Success: (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:114:124: warning: expression result unused [-Wunused-value] 114 | BPF_PRINTK("[SendEvent] Success: (type: %d, allocation address: 0x%lx, target PID: %d)\n", event->resourceType, alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:145:20: warning: left operand of comma operator has no effect [-Wunused-value] 145 | BPF_PRINTK("[ResourceFreeHelper] Failed: Getting event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:145:104: warning: left operand of comma operator has no effect [-Wunused-value] 145 | BPF_PRINTK("[ResourceFreeHelper] Failed: Getting event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:145:111: warning: expression result unused [-Wunused-value] 145 | BPF_PRINTK("[ResourceFreeHelper] Failed: Getting event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:165:20: warning: left operand of comma operator has no effect [-Wunused-value] 165 | BPF_PRINTK("[ResourceFreeHelper] Failed: Updating event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:165:105: warning: left operand of comma operator has no effect [-Wunused-value] 165 | BPF_PRINTK("[ResourceFreeHelper] Failed: Updating event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:165:112: warning: expression result unused [-Wunused-value] 165 | BPF_PRINTK("[ResourceFreeHelper] Failed: Updating event (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:169:16: warning: left operand of comma operator has no effect [-Wunused-value] 169 | BPF_PRINTK("[ResourceFreeHelper] Success: (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:169:87: warning: left operand of comma operator has no effect [-Wunused-value] 169 | BPF_PRINTK("[ResourceFreeHelper] Success: (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:169:94: warning: expression result unused [-Wunused-value] 169 | BPF_PRINTK("[ResourceFreeHelper] Success: (allocation: 0x%lx, target PID: %d)\n", alloc, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:205:20: warning: left operand of comma operator has no effect [-Wunused-value] 205 | BPF_PRINTK("[ResourceAllocHelper] Failed: Getting event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:205:110: warning: left operand of comma operator has no effect [-Wunused-value] 205 | BPF_PRINTK("[ResourceAllocHelper] Failed: Getting event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:205:116: warning: expression result unused [-Wunused-value] 205 | BPF_PRINTK("[ResourceAllocHelper] Failed: Getting event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:225:20: warning: left operand of comma operator has no effect [-Wunused-value] 225 | BPF_PRINTK("[ResourceAllocHelper] Failed: Updating event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:225:111: warning: left operand of comma operator has no effect [-Wunused-value] 225 | BPF_PRINTK("[ResourceAllocHelper] Failed: Updating event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:225:117: warning: expression result unused [-Wunused-value] 225 | BPF_PRINTK("[ResourceAllocHelper] Failed: Updating event (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:229:16: warning: left operand of comma operator has no effect [-Wunused-value] 229 | BPF_PRINTK("[ResourceAllocHelper] Success: (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:229:93: warning: left operand of comma operator has no effect [-Wunused-value] 229 | BPF_PRINTK("[ResourceAllocHelper] Success: (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~ /home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/ebpf/procdump_ebpf.c:229:99: warning: expression result unused [-Wunused-value] 229 | BPF_PRINTK("[ResourceAllocHelper] Success: (allocation size: 0x%lx, target PID: %d)\n", size, target_PID); | ^~~~~~~~~~ 33 warnings generated. [ 30%] Generating procdump.ebpf.o /usr/sbin/bpftool gen object procdump.ebpf.o procdump_ebpf.o libbpf: failed to find BTF info for object 'procdump_ebpf.o' Error: failed to link 'procdump_ebpf.o': Invalid argument (22) make[2]: *** [CMakeFiles/procdump_ebpf.dir/build.make:78: procdump.ebpf.o] Error 234 make[2]: *** Deleting file 'procdump.ebpf.o' make[2]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu' make[1]: *** [CMakeFiles/Makefile2:308: CMakeFiles/procdump_ebpf.dir/all] Error 2 make[1]: Leaving directory '/home/tkloczko/rpmbuild/BUILD/ProcDump-for-Linux-3.2.0/x86_64-redhat-linux-gnu' make: *** [Makefile:94: all] Error 2 ```

So clang (18.1.4) generates that object file and than /usr/sbin/bpftool cannot link it. What about use gcc instead clang? In this case I'm playing with gcc from Fedora rawhide which IIRC should be able to build bpf code as well 🤔 (I don't know however how to do that)

MarioHewardt commented 2 months ago

Thanks. I think GCC can build BPF code nowadays but it's not something I have tried or is supported at the moment. Feel free to try it out if you want. There are a couple of other things to check:

kloczek commented 2 months ago

As I wrote I'm using gcc from Fedora and looking on spec file I suppose as well that exactly those binaries should be able to compile bpf code and cannot only find details how to do that. Yes kernel support BFP/BTF. Will try later with libbpf-bootstrap 🤔

MarioHewardt commented 2 months ago

Not sure if this might help - https://gcc.gnu.org/wiki/BPFBackEnd

kloczek commented 2 months ago

I've saw already that page and it cannot be used in context of Fedora binaries because fedora uses multiarch gcc/binutils setup and is not using crosscompilers for bpf. If I'll not find exact solution I'll try soon to open fedora ticket with question because I have very similar issue with dtrace https://github.com/oracle/dtrace-utils/issues/71

ezannoni commented 2 months ago

If Fedora would package the crosses for BPF that would be really nice and solve all these issues indeed.

MarioHewardt commented 2 months ago

In order to support memory leak tracking (using -restrack) switch, ProcDump relies on eBPF. One option is to remove the libbpf dependencies from the build (behind a new def, NO_RESTRACK or some such). This would avoid build failures for distros without proper eBPF support. Of course, this also means that the -restrack switch would not work on those builds.

kloczek commented 2 months ago

In order to support memory leak tracking (using -restrack) switch, ProcDump relies on eBPF

That technique is known more than decade (since Solaris DTrace is known which Linux eBPF only mimics).

MarioHewardt commented 2 months ago

I'm not making any statements as to which technology came first or if one is better than the other :) Suffice to say, I was presenting an optional build strategy if you don't need memory leak tracking and hence can remove the dependency on eBPF.