libbpf-tools not building because clang attribute error

[rtkaratekid]

Hi all, I'm taking a stab at the new CO-RE API. If these are successfully compiled, they should be able to run on kernels without BTF support? Is that correct?

Regardless, here's the issue. Running Ubuntu 18.04 I just compiled and installed kernel 5.4 with BTF support (which was successful according to bpftool). I then built LLVM/Clang from source using:

apt-get install -y cmake gcc g++
git clone --depth 1 http://llvm.org/git/llvm.git
cd llvm/tools
git clone --depth 1 http://llvm.org/git/clang.git
cd ..; mkdir build; cd build;
cmake -DLLVM_TARGETS_TO_BUILD="X86;BPF" \
  -DLLVM_BUILD_LLVM_DYLIB=ON \
  -DLLVM_ENABLE_RTTI=ON \
  -DCMAKE_BUILD_TYPE=Release ..

make -j$(getconf _NPROCESSORS_ONLN)
make install

Demonstrating successful installation:

$ clang --version
clang version 10.0.0 (http://llvm.org/git/clang.git 65acf43270ea2894dffa0d0b292b92402f80c8cb) (http://llvm.org/git/llvm.git 2c4ca6832fa6b306ee6a7010bfb80a3f2596f824)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/bin

$ llc --version
LLVM (http://llvm.org/):
  LLVM version 10.0.0svn
  Optimized build.
  Default target: x86_64-unknown-linux-gnu
  Host CPU: skylake

  Registered Targets:
    bpf    - BPF (host endian)
    bpfeb  - BPF (big endian)
    bpfel  - BPF (little endian)
    x86    - 32-bit X86: Pentium-Pro and above
    x86-64 - 64-bit X86: EM64T and AMD64

I clone the BCC repo and then:

$ cd libbpf-tools/
$ make

And got this error:

In file included from drsnoop.bpf.c:3:
./vmlinux.h:5:15: error: attribute 'preserve_access_index' is not supported by '#pragma clang attribute'
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
              ^
./vmlinux.h:127438:15: error: '#pragma clang attribute pop' with no matching '#pragma clang attribute push'
#pragma clang attribute pop
              ^
2 errors generated.
Makefile:50: recipe for target '.output/drsnoop.bpf.o' failed
make: *** [.output/drsnoop.bpf.o] Error 1

I think the pop attribute issue will be solved when the push is solved. So, pulling on that thread, I went to the Clang docs since I thought the attribute issue was weird, and it looks like Clang10 does actually include preserve_access_index. Since this is an issue with the vmlinux_505.h file, I'm not really sure where to proceed from here. I don't quite understand BTF yet although today or tomorrow I'm hoping to shore up my ignorance by looking at the Facebook microsite.

Have I done something wrong, or is this an issue with the vmlinux.h generation? What can I do to fix this error?

Edit: I just realized that I'm trying to use the libbpf-tools vmlinux_505.h for a 5.4 kernel. I'll don't know how to generate my own vmlinux_504.h file yet, but I'll see if I can figure it out. That could be the issue?

Edit (again, sorry): just tried using bpftool for vmlinux generation

$ bpftool btf dump file /sys/kernel/btf/vmlinux format c

#ifndef __VMLINUX_H__
#define __VMLINUX_H__

#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
#endif

#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute pop
#endif

#endif /* __VMLINUX_H__ */

But I'm not sure how to populate the body of the vmlinux.h file.

[anakryiko]

Hi all, I'm taking a stab at the new CO-RE API. If these are successfully compiled, they should be able to run on kernels without BTF support? Is that correct?

No, to run they need target kernels to have BTF, otherwise it's impossible to relocate and adjust BPF program itself correctly.

Regardless, here's the issue. Running Ubuntu 18.04 I just compiled and installed kernel 5.4 with BTF support (which was successful according to bpftool). I then built LLVM/Clang from source using:

apt-get install -y cmake gcc g++
git clone --depth 1 http://llvm.org/git/llvm.git
cd llvm/tools
git clone --depth 1 http://llvm.org/git/clang.git

It's a real disaster the way that LLVM/Clang handled transition to single repo. Those two git repositories are stale and few people reported problems trying to use version compiled from them. Please use (now canonical) single repo: https://github.com/llvm/llvm-project Or just install Clang/LLVM 10 from packages.

Edit: I just realized that I'm trying to use the libbpf-tools vmlinux_505.h for a 5.4 kernel. I'll don't know how to generate my own vmlinux_504.h file yet, but I'll see if I can figure it out. That could be the issue?

Using 5.5 vmlinux.h with 5.4 kernel should be fine in most cases. vmlinux.h is there to provide type definitions, they don't have to be exactly the same as your target kernel. Only types that you are actually using should be compatible. If there are some new fields on 5.5, which are not present on 5.4, but your program is not accessing them - it should be ok. Which is the case almost in all cases.

Edit (again, sorry): just tried using bpftool for vmlinux generation

$ bpftool btf dump file /sys/kernel/btf/vmlinux format c

But I'm not sure how to populate the body of the vmlinux.h file.

Hm... Are you sure /sys/kernel/btf/vmlinux exists and is not empty? Check that your kernel is compiled with CONFIG_DEBUG_INFO_BTF=y.

[rtkaratekid]

@anakryiko thanks for the helpful reply.

No, to run they need target kernels to have BTF, otherwise it's impossible to relocate and adjust BPF program itself correctly.

I think that makes sense. Is this because the libbpf loader needs to be able to refer to the kernel datatypes? Even if, say, the BTF kernel is compiled for 5.5, that BTF dependent program couldn't run on a non-BTF 5.5 kernel? Sorry, I'm trying to write programs that can be easily run across one kernel version. I've tried with non-CORE BPF, but have had a lot of trouble trying to understand how to extract meaningful data from specifically tracepoints. So I'm now examining CORE to see if that's easier, but it would be a bit of a hurdle if it couldn't generalize to non-BTF supported kernels.

Is there much of a chance of distros including BTF in their default kernel build in the future?

It's a real disaster the way that LLVM/Clang handled transition to single repo. Those two git repositories are stale and few people reported problems trying to use version compiled from them. Please use (now canonical) single repo: https://github.com/llvm/llvm-project Or just install Clang/LLVM 10 from packages

This is a life saver! I had no idea that this repo was the single resource. I just compiled and installed Clang/LLVM 10 from this source. However I'm still getting the same weird attribute error:

In file included from drsnoop.bpf.c:3:
./vmlinux.h:5:15: error: attribute 'preserve_access_index' is not supported by '#pragma clang attribute'
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
              ^
./vmlinux.h:127438:15: error: '#pragma clang attribute pop' with no matching '#pragma clang attribute push'
#pragma clang attribute pop
              ^
2 errors generated.

Hm... Are you sure /sys/kernel/btf/vmlinux exists and is not empty? Check that your kernel is compiled with CONFIG_DEBUG_INFO_BTF=y.

While the above compiler error may be one of the issues, I think perhaps my BTF support wasn't configured correctly before I compiled the kernel. As you saw above, I am able to generate a vmlinux file, but it just doesn't have a body for some strange reason. I did a weird thing where I compiled the kernel with all debug options turned off (to see how fast it would be, which was actually pretty fun), and then went back and turned those options back on, as well as BTF. There's a chance I screwed up along the way and turned something off BTF needed? I'm attempting all this in a VM, so I think I'll just revert today and try it all fresh and see if it makes a difference.

As for whether it exists and if BTF support is enabled:

$ cat /sys/kernel/btf/vmlinux
��

$ ./bpftool btf dump file /sys/kernel/btf/vmlinux format c
#ifndef __VMLINUX_H__
#define __VMLINUX_H__

#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
#endif

#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute pop
#endif

#endif /* __VMLINUX_H__ */

$ cat /boot/config-5.4.0 | grep BTF
CONFIG_VIDEO_SONY_BTF_MPX=m
CONFIG_DEBUG_INFO_BTF=y

If you have any other troubleshooting ideas, I'd love to hear them. I'll definitely come back and report on if the kernel compilation was the issue.

[anakryiko]

@anakryiko thanks for the helpful reply.

No, to run they need target kernels to have BTF, otherwise it's impossible to relocate and adjust BPF program itself correctly.

I think that makes sense. Is this because the libbpf loader needs to be able to refer to the kernel datatypes?

Yes, exactly.

Even if, say, the BTF kernel is compiled for 5.5, that BTF dependent program couldn't run on a non-BTF 5.5 kernel?

Yes. While technically it is possible to run same compiled BPF code on exactly the same kernel version and kernel configuration, it's too brittle to allow w/ CO-RE. This is due to any tiny config or code change can lead to a different data type memory layout.

Sorry, I'm trying to write programs that can be easily run across one kernel version. I've tried with non-CORE BPF, but have had a lot of trouble trying to understand how to extract meaningful data from specifically tracepoints.

Tracepoints for the same kernel version should be a stable interface, so if you never need to read anything from task_struct or any other internal kernel data structure, you shouldn't need to use CO-RE actually. Start with looking at existing tracepoint-using BPF programs and how they get their data (though most of them will be using BCC, given before BPF CO-RE it was generally very hard to have non-BCC pre-compiled BPF program).

Anyways, if you can share a minimal example of code using some tracepoint and explain what you hope to achieve, people here might be able to help figure out issues you had.

So I'm now examining CORE to see if that's easier, but it would be a bit of a hurdle if it couldn't generalize to non-BTF supported kernels.

It seems like it is going to be a blocker for you.

Is there much of a chance of distros including BTF in their default kernel build in the future?

Well, it's up to users like you demanding this from distros, actually. So keep pushing :) Right now big users of FB (e.g., Facebook, and, I presume, Google, etc), are building kernels with custom configurations that enable CONFIG_DEBUG_INFO_BTF=y explicitly to be able to use it. For Linux distributions w/ stock kernel configs, there will need to be a good push from users to make BTF available by default, I'd imagine.

It's a real disaster the way that LLVM/Clang handled transition to single repo. Those two git repositories are stale and few people reported problems trying to use version compiled from them. Please use (now canonical) single repo: https://github.com/llvm/llvm-project Or just install Clang/LLVM 10 from packages

This is a life saver! I had no idea that this repo was the single resource. I just compiled and installed Clang/LLVM 10 from this source. However I'm still getting the same weird attribute error:

In file included from drsnoop.bpf.c:3:
./vmlinux.h:5:15: error: attribute 'preserve_access_index' is not supported by '#pragma clang attribute'
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
              ^
./vmlinux.h:127438:15: error: '#pragma clang attribute pop' with no matching '#pragma clang attribute push'
#pragma clang attribute pop
              ^
2 errors generated.

This is strange and probably the version of Clang that you built is not the default clang that's resolved when running just clang. Check that which clang and clang --version return point to and return version you are expecting. E.g.,:

$ clang --version
clang version 10.0.0 (https://github.com/llvm/llvm-project.git d32170dbd5b0d54436537b6b75beaf44324e0c28)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/andriin/bin

I just built llvmorg-10.0.0 tag from llvm-project and it successfully compiled libbpf-tools from BCC. Alternatively, you can explicitly specify path to clang binary to use when building libbpf-tools:

$ pwd
/home/andriin/local/bcc/libbpf-tools
$ make clean && make CLANG=/path/to/your/clang-10 -j30
... <success> ...

Hm... Are you sure /sys/kernel/btf/vmlinux exists and is not empty? Check that your kernel is compiled with CONFIG_DEBUG_INFO_BTF=y.

While the above compiler error may be one of the issues, I think perhaps my BTF support wasn't configured correctly before I compiled the kernel. As you saw above, I am able to generate a vmlinux file, but it just doesn't have a body for some strange reason. I did a weird thing where I compiled the kernel with all debug options turned off (to see how fast it would be, which was actually pretty fun), and then went back and turned those options back on, as well as BTF. There's a chance I screwed up along the way and turned something off BTF needed? I'm attempting all this in a VM, so I think I'll just revert today and try it all fresh and see if it makes a difference.

I don't know and it's hard to tell. Verify that /sys/kernel/btf/vmlinux is non-empty file. You should also see a lot of output when running in non-"format c" mode with bpftool:

$ ls -la /sys/kernel/btf/vmlinux
-r--r--r--. 1 root root 3490473 Apr  5 06:41 /sys/kernel/btf/vmlinux
$ bpftool btf dump file /sys/kernel/btf/vmlinux | head -n10
[1] INT '(anon)' size=4 bits_offset=0 nr_bits=32 encoding=(none)
[2] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
[3] CONST '(anon)' type_id=2
[4] VOLATILE '(anon)' type_id=2
[5] ARRAY '(anon)' type_id=2 index_type_id=1 nr_elems=2
[6] PTR '(anon)' type_id=9
[7] CONST '(anon)' type_id=6
[8] INT 'char' size=1 bits_offset=0 nr_bits=8 encoding=(none)
[9] CONST '(anon)' type_id=8
[10] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)
$ bpftool btf dump file /sys/kernel/btf/vmlinux format c | head -n20
#ifndef __VMLINUX_H__
#define __VMLINUX_H__

#ifndef BPF_NO_PRESERVE_ACCESS_INDEX
#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
#endif

typedef signed char __s8;

typedef unsigned char __u8;

typedef short int __s16;

typedef short unsigned int __u16;

typedef int __s32;

typedef unsigned int __u32;

typedef long long int __s64;

$ cat /boot/config-5.4.0 | grep BTF CONFIG_VIDEO_SONY_BTF_MPX=m CONFIG_DEBUG_INFO_BTF=y

If you have any other troubleshooting ideas, I'd love to hear them. I'll definitely come back and report on if the kernel compilation was the issue.

Not really, just make sure you have recent enough bpftool, but given it outputs correct preamble, I think it's recent enough already... So just double-check all the configuration. BTW, to check config, I'd zcat procfs entry for Kconfig instead (just having config file in /boot doesn't mean that you booted correct kernel). Here's what I have:

$ zcat /proc/config.gz | grep DEBUG_INFO
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_INFO_REDUCED is not set
# CONFIG_DEBUG_INFO_SPLIT is not set
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_DEBUG_INFO_BTF=y

[rtkaratekid]

Okay took a while, but I finally got back to this.

Turns out the BTF issue above was due to some weird kernel compilation stuff. I'm doing all this in a VM so when I just destroyed and rebuilt the VM and compiled the kernel with BTF support, those issues figured themselves out. I was able to generate what looks like a valid vmlinux.h file for kernel 5.4.

At that point I apt install clang (I'm on Ubuntu 18.04) and got version 6.0. Tried building libbpf-tools and predictably got the attribute error also mentioned above.

Went to the LLVM site and ran their install script # bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" to get clang-10. Reset the sym link in /usr/bin to point to clang-10.

Went right back to libbpf-tools. Ran make, and didn't get the attribute error or something about BTF (yet) but I found a new error:

vagrant@ubuntu1804:~/bcc/libbpf-tools$ make clean && make
  CLEAN
  MKDIR    .output
  MKDIR    libbpf
  LIB      libbpf.a
mkdir -p /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c bpf.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/bpf.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c btf.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/btf.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c libbpf.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c libbpf_errno.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf_errno.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c netlink.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/netlink.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c nlattr.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/nlattr.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c str_error.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/str_error.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c libbpf_probes.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf_probes.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c bpf_prog_linfo.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/bpf_prog_linfo.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c xsk.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/xsk.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c btf_dump.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/btf_dump.o
cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c hashmap.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/hashmap.o
ar rcs /home/vagrant/bcc/libbpf-tools/.output//libbpf/libbpf.a /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/bpf.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/btf.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf_errno.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/netlink.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/nlattr.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/str_error.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf_probes.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/bpf_prog_linfo.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/xsk.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/btf_dump.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/hashmap.o
sed -e "s|@PREFIX@|/usr|" \
    -e "s|@LIBDIR@||" \
    -e "s|@VERSION@|0.0.8|" \
    < libbpf.pc.template > /home/vagrant/bcc/libbpf-tools/.output//libbpf/libbpf.pc
if [ ! -d '/home/vagrant/bcc/libbpf-tools/.output//bpf' ]; then install -d -m 755 '/home/vagrant/bcc/libbpf-tools/.output//bpf'; fi; install bpf.h libbpf.h btf.h xsk.h libbpf_util.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h libbpf_common.h -m 644 '/home/vagrant/bcc/libbpf-tools/.output//bpf'
if [ ! -d '/home/vagrant/bcc/libbpf-tools/.output//pkgconfig' ]; then install -d -m 755 '/home/vagrant/bcc/libbpf-tools/.output//pkgconfig'; fi; install /home/vagrant/bcc/libbpf-tools/.output//libbpf/libbpf.pc -m 644 '/home/vagrant/bcc/libbpf-tools/.output//pkgconfig'
if [ ! -d '/home/vagrant/bcc/libbpf-tools/.output/' ]; then install -d -m 755 '/home/vagrant/bcc/libbpf-tools/.output/'; fi; cp -fpR /home/vagrant/bcc/libbpf-tools/.output//libbpf/libbpf.a  '/home/vagrant/bcc/libbpf-tools/.output/'
  BPF      drsnoop.bpf.o
drsnoop.bpf.c:85:38: error: use of undeclared identifier 'BPF_F_CURRENT_CPU'
        bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,
                                            ^
1 error generated.
Makefile:50: recipe for target '.output/drsnoop.bpf.o' failed
make: *** [.output/drsnoop.bpf.o] Error 1

I'm not sure what's going on here because it looks like, in the makefile, we are being sure to include libbpf/uapi/linux/bpf.h which is where it looks like BPF_F_CURRENT_CPU lives. Has this issue come up before? Is there some sort of configuration step that I'm missing here?

Still excited about contributing to the new API, just need to get it building.

[anakryiko]

On Fri, May 1, 2020 at 7:07 AM rtkaratekid notifications@github.com wrote:

Okay took a while, but I finally got back to this.

Turns out the BTF issue above was due to some weird kernel compilation stuff. I'm doing all this in a VM so when I just destroyed and rebuilt the VM and compiled the kernel with BTF support, those issues figured themselves out. I was able to generate what looks like a valid vmlinux.h file for kernel 5.4.

Sorry it was so painful, but I'm glad you figured it out!

At that point I apt install clang (I'm on Ubuntu 18.04) and got version 6.0. Tried building libbpf-tools and predictably got the attribute error also mentioned above.

Went to the LLVM site and ran their install script # bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)" to get clang-10. Reset the sym link in /usr/bin to point to clang-10.

Went right back to libbpf-tools. Ran make, and didn't get the attribute error or something about BTF (yet) but I found a new error:

vagrant@ubuntu1804:~/bcc/libbpf-tools$ make clean && make CLEAN MKDIR .output MKDIR libbpf LIB libbpf.a mkdir -p /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c bpf.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/bpf.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c btf.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/btf.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c libbpf.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c libbpf_errno.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf_errno.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c netlink.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/netlink.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c nlattr.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/nlattr.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c str_error.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/str_error.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c libbpf_probes.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf_probes.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c bpf_prog_linfo.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/bpf_prog_linfo.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c xsk.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/xsk.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c btf_dump.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/btf_dump.o cc -I. -I../include -I../include/uapi -g -O2 -Werror -Wall -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -c hashmap.c -o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/hashmap.o ar rcs /home/vagrant/bcc/libbpf-tools/.output//libbpf/libbpf.a /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/bpf.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/btf.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf_errno.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/netlink.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/nlattr.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/str_error.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/libbpf_probes.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/bpf_prog_linfo.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/xsk.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/btf_dump.o /home/vagrant/bcc/libbpf-tools/.output//libbpf/staticobjs/hashmap.o sed -e "s|@PREFIX@|/usr|" \ -e "s|@LIBDIR@||" \ -e "s|@VERSION@|0.0.8|" \ < libbpf.pc.template > /home/vagrant/bcc/libbpf-tools/.output//libbpf/libbpf.pc if [ ! -d '/home/vagrant/bcc/libbpf-tools/.output//bpf' ]; then install -d -m 755 '/home/vagrant/bcc/libbpf-tools/.output//bpf'; fi; install bpf.h libbpf.h btf.h xsk.h libbpf_util.h bpf_helpers.h bpf_helper_defs.h bpf_tracing.h bpf_endian.h bpf_core_read.h libbpf_common.h -m 644 '/home/vagrant/bcc/libbpf-tools/.output//bpf' if [ ! -d '/home/vagrant/bcc/libbpf-tools/.output//pkgconfig' ]; then install -d -m 755 '/home/vagrant/bcc/libbpf-tools/.output//pkgconfig'; fi; install /home/vagrant/bcc/libbpf-tools/.output//libbpf/libbpf.pc -m 644 '/home/vagrant/bcc/libbpf-tools/.output//pkgconfig' if [ ! -d '/home/vagrant/bcc/libbpf-tools/.output/' ]; then install -d -m 755 '/home/vagrant/bcc/libbpf-tools/.output/'; fi; cp -fpR /home/vagrant/bcc/libbpf-tools/.output//libbpf/libbpf.a '/home/vagrant/bcc/libbpf-tools/.output/' BPF drsnoop.bpf.o drsnoop.bpf.c:85:38: error: use of undeclared identifier 'BPF_F_CURRENT_CPU' bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,

This is super strangle. Check libbpf-tools/vmlinux.h. It is included from drsnoop.bpf.c. It has definition of BPF_F_CURRENT_CPU:

BPF_F_CURRENT_CPU = 4294967295,

So I'm not sure what's going on. None of libbpf-tolls include libbpf/uapi/linux/bpf.h, btw, so that shouldn't be relevant.

One next step to investigate this would be to run make in verbose mode:

$ make V=1

I see this invocation for drsnoop.bpf.c:

clang -g -O2 -target bpf -D__TARGET_ARCH_x86 \ -I.output -c drsnoop.bpf.c -o .output/drsnoop.bpf.o && \ llvm-strip -g .output/drsnoop.bpf.o

If you have the same, I'd try to add -I. to the list of include paths to ensure that vmlinux.h is definitely picked up (but I think it should be already...).

One more thing, just in case, try replacing #include "vmlinux.h" with

include "vmlinux_505.h", see if that might help. Again, it shouldn't

matter, but who knows.

Other than that, not sure what else I can suggest at this point, sorry!

                                        ^

1 error generated. Makefile:50: recipe for target '.output/drsnoop.bpf.o' failed make: *** [.output/drsnoop.bpf.o] Error 1

I'm not sure what's going on here because it looks like, in the makefile, we are being sure to include libbpf/uapi/linux/bpf.h which is where it looks like BPF_F_CURRENT_CPU lives. Has this issue come up before? Is there some sort of configuration step that I'm missing here?

Still excited about contributing to the new API, just need to get it building.

I know it's really frustrating, but hopefully we'll figure it out and if something is wrong with the way libbpf-tools are set up, we'll fix it and it will be better for the next person!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/iovisor/bcc/issues/2855#issuecomment-622402539, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4BK3BJIDUEK4V2DEBOETRPLJTLANCNFSM4L22YXXQ .

[rtkaratekid]

Thanks for the reply @anakryiko!

I just want to come clean for a second here, and confess to you I'm also the annoying guy who's been sending questions on the iovisor-dev mailing list. Hopefully I'm not as annoying as I feel, but I'm determined to get this stuff working and you've been really great and gracious with your time to help me out! So again, thanks! (I am still also learning how to do non-CO-RE BPF as well, with very small degrees of success. Still working on getting data from tracepoints and kprobes and coming up with a sure-fire build system.)

One more thing, just in case, try replacing #include "vmlinux.h" with

include "vmlinux_505.h", see if that might help. Again, it shouldn't

matter, but who knows.

Okay, since that was the lowest effort thing to do (sorry, it's just friday), I tried that first. It failed at first because llvm-strip was llvm-strip-9. I symlinked to that and the build was successful using vmlinux_505.h!

So that's great, but now, when running any of the tools, e.g. # ./opensnoop I get these errors:

libbpf: Error in bpf_object__probe_name():Function not implemented(38). Couldn't load basic 'r0 = 0' BPF program.
libbpf: Error in bpf_object__probe_global_data():Function not implemented(38). Couldn't create simple array map.
libbpf: Error loading BTF: Function not implemented(38)
libbpf: Error loading .BTF into kernel: -38.
libbpf: kernel doesn't support global data
libbpf: failed to load object 'opensnoop_bpf'
libbpf: failed to load BPF skeleton 'opensnoop_bpf': -95
failed to load BPF object: -95

This makes me think that BTF is actually not supported by my kernel... So I decided to check:

This puts out what looks like a full and valid vmlinux file
$ bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

The config here looks fine
$ cat /boot/config-5.4.0 | grep BTF
CONFIG_DEBUG_INFO_BTF=y

$ zcat /proc/config.gz | grep DEBUG_INFO

Unfortunately I didn't have CONFIG_IKCONFIG or CONFIG_IKCONFIG_PROC set so the last option wasn't helpful.

At this point, I'm hesitant to say, but it looks like kernel 5.4 isn't having it. The kernel seems to be built correctly, but doesn't support running the libbpf tools in the repo. Does this mean that these tools will only work on Linux 5.5+?

Kernel 5.4 seems to not generate the macro BPF_F_CURRENT_CPU (I've tried a few times) and it can build the tools with vmlinux_505.h, but it won't run them. Hopefully this is helpful. I'm scratching my head over this as well. Perhaps I'll try this over again with Linux 5.5 and see how it goes.

[anakryiko]

On Fri, May 1, 2020 at 1:59 PM rtkaratekid notifications@github.com wrote:

Thanks for the reply @anakryiko https://github.com/anakryiko!

I just want to come clean for a second here, and confess to you I'm also the annoying guy who's been sending questions on the iovisor-dev mailing list. Hopefully I'm not as annoying as I feel, but I'm determined to get this stuff working and you've been really great and gracious with your time to help me out! So again, thanks! (I am still also learning how to do non-CO-RE BPF as well, with very small degrees of success. Still working on getting data from tracepoints and kprobes and coming up with a sure-fire build system.)

Not at all, I hope you'll finally will get to the point where it's not a struggle to build and run everything and can finally play around with exciting BPF features.

One more thing, just in case, try replacing #include "vmlinux.h" with

include "vmlinux_505.h", see if that might help. Again, it shouldn't

matter, but who knows.

Okay, since that was the lowest effort thing to do (sorry, it's just friday), I tried that first. It failed at first because llvm-strip was llvm-strip-9. I symlinked to that and the build was successful using vmlinux_505.h!

Well, that's interesting, because vmlinux.h is symlink to vmlinux_505.h, so if vmlinux.h doesn't work, but vmlinux_505.h, then something is very weird on your side.

So that's great, but now, when running any of the tools, e.g. # ./opensnoop I get these errors:

libbpf: Error in bpf_object__probe_name():Function not implemented(38). Couldn't load basic 'r0 = 0' BPF program.

Couldn't load basic 'r0 = 0' program suggests that:

• either RLIMIT_MEMLOCK wasn't bumped. In this case tools usually output "failed to increase rlimit". Check if you see that in your verbose logs?
• your kernel is not configured to have BPF enabled. Check your kernel config, here's all the settings with BPF on my test system:

$ grep BPF ~/linux-build/default/.config

        CONFIG_CGROUP_BPF=y

CONFIG_BPF=y CONFIG_BPF_LSM=y CONFIG_BPF_SYSCALL=y CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y

CONFIG_BPF_JIT_ALWAYS_ON is not set

CONFIG_BPF_JIT_DEFAULT_ON=y CONFIG_IPV6_SEG6_BPF=y CONFIG_NETFILTER_XT_MATCH_BPF=y CONFIG_BPFILTER=y CONFIG_BPFILTER_UMH=m CONFIG_NET_CLS_BPF=y CONFIG_NET_ACT_BPF=y CONFIG_BPF_JIT=y CONFIG_BPF_STREAM_PARSER=y CONFIG_LWTUNNEL_BPF=y CONFIG_HAVE_EBPF_JIT=y CONFIG_BPF_EVENTS=y CONFIG_BPF_KPROBE_OVERRIDE=y CONFIG_TEST_BPF=m

You'll need at least CONFIG_BPF=y and CONFIG_BPF_SYSCALL=y to get started.

libbpf: Error in bpf_object__probe_global_data():Function not implemented(38). Couldn't create simple array map. libbpf: Error loading BTF: Function not implemented(38) libbpf: Error loading .BTF into kernel: -38. libbpf: kernel doesn't support global data libbpf: failed to load object 'opensnoop_bpf' libbpf: failed to load BPF skeleton 'opensnoop_bpf': -95 failed to load BPF object: -95

This makes me think that BTF is actually not supported by my kernel... So I decided to check:

Not really. libbpf failed to "upload" program's BTF into kernel. But again, it's due to no BPF support, probably. bpftool dump below clearly shows that you do have BTF enabled and built, but it's a bit orthogonal to BPF. You can have BTF, but disabled BPF.

This puts out what looks like a full and valid vmlinux file $ bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

The config here looks fine $ cat /boot/config-5.4.0 | grep BTF CONFIG_DEBUG_INFO_BTF=y

$ zcat /proc/config.gz | grep DEBUG_INFO

Unfortunately I didn't have CONFIG_IKCONFIG or CONFIG_IKCONFIG_PROC set so the last option wasn't helpful.

At this point, I'm hesitant to say, but it looks like kernel 5.4 isn't having it. The kernel seems to be built correctly, but doesn't support running the libbpf tools in the repo. Does this mean that these tools will only work on Linux 5.5+?

Kernel 5.4 seems to not generate the macro BPF_F_CURRENT_CPU (I've tried a few times) and it can build the tools with vmlinux_505.h, but it won't run them. Hopefully this is helpful. I'm scratching my head over this as well. Perhaps I'll try this over again with Linux 5.5 and see how it goes.

5.4 kernel might not enable BPF_F_CURRENT_CPU generation in vmlinux.h (only one of the more recent releases record BPF_F_CURRENT_CPU as enum), but it shouldn't be a problem given you use pre-generated vmlinux.h. But overall 5.4 should be plenty recent for a lot of BPF functionality.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/iovisor/bcc/issues/2855#issuecomment-622565981, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAD4BK442ZMM4SFM3UCAW5LRPMZ3PANCNFSM4L22YXXQ .

[CentUser]

I run into this problem too. After reading all your posts, seems like it is caused by the clang compiler. I used centos 8 app-stream clang version:8.0.1, still get these errors. I'll try to install the latest llvm.

[inevity]

Ubuntu Jammy Jellyfish (development branch), use clang 14 from apt.llvm.org, build ok. Maybe use need use the bpftool from linux-common-tools , not the libtools/bin/bpftool.