Build fails with libbpf > 0.8.x, can only build kernel when using exact submodule commit

[Apteryks]

The build fails like:

[...]
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c: In function ‘create_new_enumeration64’:
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:350:54: error: invalid use of undefined type ‘struct btf_enum64’
  350 |                 const char *name = cu__btf_str(cu, ep[i].name_off);
      |                                                      ^
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:350:57: error: invalid use of undefined type ‘struct btf_enum64’
  350 |                 const char *name = cu__btf_str(cu, ep[i].name_off);
      |                                                         ^
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:351:54: error: invalid use of undefined type ‘struct btf_enum64’
  351 |                 uint64_t value = btf_enum64_value(&ep[i]);
      |                                                      ^
make[2]: *** [CMakeFiles/dwarves.dir/build.make:163: CMakeFiles/dwarves.dir/btf_loader.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make[1]: *** [CMakeFiles/Makefile2:110: CMakeFiles/dwarves.dir/all] Error 2
make[1]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make: *** [Makefile:139: all] Error 2

[Apteryks]

I've reproduced the above using the latest available commit, d5847615d0a5c082b1d61bdefc4683109c49de67.

[Apteryks]

I had to use libbpf 0.8.3.

[Apteryks]

Although it built fine, compiling the Linux kernel with CONFIG_DEBUG_INFO would fail with a failed to find '.BTF' ELF section in vmlinux error message; using the exact same commit as the libbpf submodule was needed.

It seems dwarves is very tightly coupled to the inner workings of libbpf? It'd be nice if it targetted at least a released version of libbpf.

[martinetd]

nixpkgs builds pahole with libbpf v1.4.1, so either of us are doing something wrong - please provide a reproducer or explain how you're building

$ ldd /nix/store/j6yi4ngcwf423igmqspjdxvkw7g3mvgn-pahole-1.26/lib/libdwarves.so.1.0.0 | grep bpf
    libbpf.so.1 => /nix/store/bbj6lrp82h79viq74pp8fzwdbag8l80n-libbpf-1.4.1/lib/libbpf.so.1 (0x00007fc14afea000)

[Apteryks]

Hi! It's fairly easy to reproduce for me, using libbpf v1.4.1, ensuring I don't pull the git submodules of dwarves, specifying the "-DLIBBPF_EMBEDDED=OFF" build option and providing pkg-config as a build input:

starting phase `configure'
source directory: "/tmp/guix-build-dwarves-1.26.drv-0/source" (relative from build: "../source")
build directory: "/tmp/guix-build-dwarves-1.26.drv-0/build"
running 'cmake' with arguments ("../source" "-DCMAKE_BUILD_TYPE=RelWithDebInfo" "-DCMAKE_INSTALL_PREFIX=/gnu/store/xgnss5xh7vhjwxl5mbmcpar4kzhfbdl8-dwarves-1.26" "-DCMAKE_INSTALL_LIBDIR=lib" "-DCMAKE_INSTALL_RPATH_USE_LINK_PATH=TRUE" "-DCMAKE_INSTALL_RPATH=/gnu/store/xgnss5xh7vhjwxl5mbmcpar4kzhfbdl8-dwarves-1.26/lib" "-DCMAKE_VERBOSE_MAKEFILE=ON" "-D__LIB=lib" "-DLIBBPF_EMBEDDED=OFF")
-- The C compiler identification is GNU 11.3.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PkgConfig: /gnu/store/jz5dwdxq4di29cd0rjjzkw356dhkzjil-pkg-config-0.29.2/bin/pkg-config (found version "0.29.2") 
-- Checking for module 'libbpf>=0.4.0'
--   Found libbpf, version 1.4.1
-- Setting BUILD_SHARED_LIBS = ON
-- Checking availability of DWARF and ELF development libraries
-- Looking for dwfl_module_build_id in elf
-- Looking for dwfl_module_build_id in elf - found
-- Found dwarf.h header: /gnu/store/iyaad74kw54jrqzpwm5r4jagfr8dgirp-elfutils-0.187/include
-- Found elfutils/libdw.h header: /gnu/store/iyaad74kw54jrqzpwm5r4jagfr8dgirp-elfutils-0.187/include
-- Found libdw library: /gnu/store/iyaad74kw54jrqzpwm5r4jagfr8dgirp-elfutils-0.187/lib/libdw.so
-- Found libelf library: /gnu/store/iyaad74kw54jrqzpwm5r4jagfr8dgirp-elfutils-0.187/lib/libelf.so
-- Checking availability of DWARF and ELF development libraries - done
-- Found ZLIB: /gnu/store/slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13/lib/libz.so (found version "1.2.13") 
-- Checking availability of argp library
-- Assuming argp is in libc
-- Checking availability of argp library - done
-- Checking availability of obstack library
-- Assuming obstack is in libc
-- Checking availability of obstack library - done
-- Performing Test HAVE_REALLOCARRAY_SUPPORT
-- Performing Test HAVE_REALLOCARRAY_SUPPORT - Success
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    CMAKE_INSTALL_LIBDIR

-- Build files have been written to: /tmp/guix-build-dwarves-1.26.drv-0/build
phase `configure' succeeded after 1.8 seconds
starting phase `patch-generated-file-shebangs'
phase `patch-generated-file-shebangs' succeeded after 0.0 seconds
starting phase `build'
/gnu/store/gl26kr5v6ch5lc3ignly61kb224drijc-cmake-minimal-3.24.2/bin/cmake -S/tmp/guix-build-dwarves-1.26.drv-0/source -B/tmp/guix-build-dwarves-1.26.drv-0/build --check-build-system CMakeFiles/Makefile.cmake 0
/gnu/store/gl26kr5v6ch5lc3ignly61kb224drijc-cmake-minimal-3.24.2/bin/cmake -E cmake_progress_start /tmp/guix-build-dwarves-1.26.drv-0/build/CMakeFiles /tmp/guix-build-dwarves-1.26.drv-0/build//CMakeFiles/progress.marks
make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make  -f CMakeFiles/dwarves.dir/build.make CMakeFiles/dwarves.dir/depend
make[2]: Entering directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
cd /tmp/guix-build-dwarves-1.26.drv-0/build && /gnu/store/gl26kr5v6ch5lc3ignly61kb224drijc-cmake-minimal-3.24.2/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/guix-build-dwarves-1.26.drv-0/source /tmp/guix-build-dwarves-1.26.drv-0/source /tmp/guix-build-dwarves-1.26.drv-0/build /tmp/guix-build-dwarves-1.26.drv-0/build /tmp/guix-build-dwarves-1.26.drv-0/build/CMakeFiles/dwarves.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make  -f CMakeFiles/dwarves.dir/build.make CMakeFiles/dwarves.dir/build
make[2]: Entering directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
[  2%] Building C object CMakeFiles/dwarves.dir/dwarves.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/dwarves.c.o -MF CMakeFiles/dwarves.dir/dwarves.c.o.d -o CMakeFiles/dwarves.dir/dwarves.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/dwarves.c
[  5%] Building C object CMakeFiles/dwarves.dir/gobuffer.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/gobuffer.c.o -MF CMakeFiles/dwarves.dir/gobuffer.c.o.d -o CMakeFiles/dwarves.dir/gobuffer.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/gobuffer.c
[  8%] Building C object CMakeFiles/dwarves.dir/dwarves_fprintf.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/dwarves_fprintf.c.o -MF CMakeFiles/dwarves.dir/dwarves_fprintf.c.o.d -o CMakeFiles/dwarves.dir/dwarves_fprintf.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/dwarves_fprintf.c
[ 10%] Building C object CMakeFiles/dwarves.dir/ctf_loader.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/ctf_loader.c.o -MF CMakeFiles/dwarves.dir/ctf_loader.c.o.d -o CMakeFiles/dwarves.dir/ctf_loader.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/ctf_loader.c
[ 13%] Building C object CMakeFiles/dwarves.dir/libctf.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/libctf.c.o -MF CMakeFiles/dwarves.dir/libctf.c.o.d -o CMakeFiles/dwarves.dir/libctf.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/libctf.c
[ 16%] Building C object CMakeFiles/dwarves.dir/btf_encoder.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/btf_encoder.c.o -MF CMakeFiles/dwarves.dir/btf_encoder.c.o.d -o CMakeFiles/dwarves.dir/btf_encoder.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/btf_encoder.c
[ 18%] Building C object CMakeFiles/dwarves.dir/btf_loader.c.o
/gnu/store/5lqhcv91ijy82p92ac6g5xw48l0lwwz4-gcc-11.3.0/bin/gcc -DDWARVES_MAJOR_VERSION=1 -DDWARVES_MINOR_VERSION=26 -D_GNU_SOURCE -Ddwarves_EXPORTS -I/tmp/guix-build-dwarves-1.26.drv-0/build -I/tmp/guix-build-dwarves-1.26.drv-0/source -pthread -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -g -DNDEBUG -fPIC -MD -MT CMakeFiles/dwarves.dir/btf_loader.c.o -MF CMakeFiles/dwarves.dir/btf_loader.c.o.d -o CMakeFiles/dwarves.dir/btf_loader.c.o -c /tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c: In function create_new_enumeration64:
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:350:54: error: invalid use of undefined type struct btf_enum64
  350 |                 const char *name = cu__btf_str(cu, ep[i].name_off);
      |                                                      ^
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:350:57: error: invalid use of undefined type struct btf_enum64
  350 |                 const char *name = cu__btf_str(cu, ep[i].name_off);
      |                                                         ^
/tmp/guix-build-dwarves-1.26.drv-0/source/btf_loader.c:351:54: error: invalid use of undefined type struct btf_enum64
  351 |                 uint64_t value = btf_enum64_value(&ep[i]);
      |                                                      ^
make[2]: *** [CMakeFiles/dwarves.dir/build.make:163: CMakeFiles/dwarves.dir/btf_loader.c.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make[1]: *** [CMakeFiles/Makefile2:110: CMakeFiles/dwarves.dir/all] Error 2
make[1]: Leaving directory '/tmp/guix-build-dwarves-1.26.drv-0/build'
make: *** [Makefile:139: all] Error 2
error: in phase 'build': uncaught exception:
%exception #<&invoke-error program: "make" arguments: ("-j" "4") exit-status: 2 term-signal: #f stop-signal: #f> 
phase `build' failed after 1.7 seconds
command "make" "-j" "4" failed with status 2
builder for `/gnu/store/b3rzaqikjqn2sf1m3m1lb1gqrc6kgybv-dwarves-1.26.drv' failed with exit code 1
build of /gnu/store/b3rzaqikjqn2sf1m3m1lb1gqrc6kgybv-dwarves-1.26.drv failed
View build log at '/var/log/guix/drvs/b3/rzaqikjqn2sf1m3m1lb1gqrc6kgybv-dwarves-1.26.drv'.
guix build: error: build of `/gnu/store/b3rzaqikjqn2sf1m3m1lb1gqrc6kgybv-dwarves-1.26.drv' failed

[Apteryks]

I note that pahole on nixpkgs use a different git repo snapshot: https://git.kernel.org/pub/scm/devel/pahole/pahole.git/snapshot/pahole-1.26.tar.gz to build it, though it fails the same for me if I use guix build dwarves --with-source=dwarves=https://git.kernel.org/pub/scm/devel/pahole/pahole.git/snapshot/pahole-1.26.tar.gz.

[Apteryks]

I found the underlying issue: https://github.com/libbpf/libbpf/issues/562

[Apteryks]

I note that the default headers used in the build environment on Guix are at 5.15.49, while Nix uses 6.7, which is probably why it works there.

[Apteryks]

If I add the headers of 6.8 to the build environment, it builds fine, so it seems the #562 fix of libbpf is insufficient, at least for kernel headers as old as 5.15.49.

[Apteryks]

Fails as well on 5.15.157. I'll report that the fix doesn't work for 5.15.X on the libbpf issue.

[martinetd]

If I add the headers of 6.8 to the build environment, it build fine, so it seems the #562 fix of libbpf is insufficient, at least kernel headers as old as 5.15.49.

It's a bit worse than that, for bpf-related headers we install the headers that were shipped in libbpf, so we always build with whatever is compatible with libbpf (this is actually a bit of a hack as I don't see where we give it priority over linux headers... but it seems to work out in practice) without worrying about what kernel includes are used:

  postInstall = ''
    # install linux's libbpf-compatible linux/btf.h
    install -Dm444 include/uapi/linux/*.h -t $out/include/linux
  '';

(We do this because we also had quite a few problems with older linux headers in the past, as some libbpf releases depended on non-released kernels. using newer libbpf headers is fine as kernel headers are guaranteed to be backwards compatible.)

Either way it's not a dwarves problem, you probably cannot build anything with libbpf in that configuration (newer libbpf and older linux/bpf.h), so this probably should be closed in favor of a new issue in libbpf.

[Apteryks]

I've pinged the libbpf people at: https://github.com/libbpf/libbpf/issues/562#issuecomment-2104544471.

[martinetd]

Sorry, I should have checked the actual error here before speaking, it's not really a libbpf problem.

libbpf 1.x provides APIs that use btf_enum64, but the struct btf_enum64 definition itself comes from the linux uapi headers, so libbpf/libbpf#562 rightfully made libbpf's btf.h not depend on the struct definition (so code that doesn't use these functions can build with older kernel heades), but using the functions themselves require manipulating the structs so the pahole code needs the definition.

I guess pahole could add a check for it and either skip the code or redefine the enum, but there's no easy define to check for this so frankly you'll be better off upgrading your linux headers or using libbpf's uapi files like nix does...

[Apteryks]

Hm, upgrading the linux-kernel-headers will have to wait until the next core updates iteration for Guix. Meanwhile copying the whole Linux 6.8 headers would add 6.8 MiB to the size of libbpf. Not great. Perhaps I could do with just bpf.h and types.h and a few others...

[Apteryks]

Also, what is uapi ? I don't see this directory in the headers of our linux-libre-headers package.

[Apteryks]

I've added the following phase to libbpf, which emulates what's done in Nix using the least amount of Linux headers:

(add-after 'install 'install-linux-bpf-headers
            ;; Workaround users such as 'dwarves' requiring btf_enum64
            ;; definition from the kernel Linux >= 6 headers (see:
            ;; https://github.com/acmel/dwarves/issues/49).
            ;; TODO: Remove once our 'linux-libre-headers' package is
            ;; upgraded to a >= 6 release.
            (lambda _
              (let ((linux-libre-headers #$(this-package-native-input
                                            "linux-libre-headers")))
                (for-each (lambda (f)
                            (install-file (string-append linux-libre-headers
                                                         "/include/" f)
                                          (string-append #$output "/include/"
                                                         (dirname f))))
                          ;; This list contains btf.h and its transitive
                          ;; dependencies.
                          (list "asm/posix_types.h"
                                "asm/types.h"
                                "asm-generic/types.h"
                                "asm-generic/int-ll64.h"
                                "linux/btf.h"
                                "linux/posix_types.h"
                                "linux/stddef.h"
                                "linux/types.h")))))

It adds about 100 KiB of overhead, and appears to resolve the issues.