search

solo-io / bumblebee

Get eBPF programs running from the cloud to the kernel in 1 line of bash
Apache License 2.0
1.26k stars 78 forks source link

ERROR Loading BPF program and maps into Kernel #44

Open nickschuetz opened 2 years ago

nickschuetz commented 2 years ago

Note that this is a VM on a Mac M1 running Ubuntu 21.10 (GNU/Linux 5.13.0-23-generic aarch64). I compiled this bee binary on the same vm it's running on.

$ file /usr/local/bin/bee
/usr/local/bin/bee: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=7d6I66htAooEyvvjafcO/4smkmG1QZgmhRnqbyK1N/AcgWG-x3EfW5StWmnIUX/_YCuHdBV4WDMGPDjTZML, not stripped
sudo bee run ghcr.io/solo-io/bumblebee/tcpconnect:0.0.9
 SUCCESS  Fetching program from registry: ghcr.io/solo-io/bumblebee/tcpconnect:0.0.9
  ERROR   Loading BPF program and maps into Kernel
Error: program tcp_v4_connect: load program: invalid argument: trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.
R1 type=ctx expected=fp
; int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
0: (85) call unknown#195896080
invalid func unknown#195896080
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
2022/01/07 00:17:28 exiting: program tcp_v4_connect: load program: invalid argument: trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.
R1 type=ctx expected=fp
; int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
0: (85) call unknown#195896080
invalid func unknown#195896080
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

This same example runs on amd64 w/o issue.

yuval-k commented 2 years ago

weird, works on my linux (5.15.11-200.fc35.x86_64)... could be an arm thing? are the other probes working?

lgadban commented 2 years ago

reposting here for posterity, can you try removing the BPF_F_NO_PREALLOC field here: https://github.com/solo-io/bumblebee/blob/main/examples/tcpconnect/tcpconnect.c#L20

and then rebuild the probe locally?

nickschuetz commented 2 years ago

Same result:

# bee run tcpconnect
 SUCCESS  Fetching program from registry: tcpconnect
  ERROR   Loading BPF program and maps into Kernel
Error: program tcp_v4_connect_ret: load program: invalid argument: R1 type=ctx expected=fp
; int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
0: (85) call unknown#195896080
invalid func unknown#195896080
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
2022/01/10 17:23:12 exiting: program tcp_v4_connect_ret: load program: invalid argument: R1 type=ctx expected=fp
; int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
0: (85) call unknown#195896080
invalid func unknown#195896080
processed 1 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
nickschuetz commented 2 years ago

Thanks @lgadban for pointing this out: https://nakryiko.com/posts/bpf-core-reference-guide/#guarding-potentially-failing-relocations

nickschuetz commented 2 years ago

After including vmlinux_arm.h:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux_arm.h

and attempting a build I get:

# ./builder/build.sh ./examples/tcpconnect/tcpconnect.c tcpconnect.o
./examples/tcpconnect/tcpconnect.c:107:5: error: no member named 'di' in 'struct pt_regs'
int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:365:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
<scratch space>:42:1: note: expanded from here
___bpf_kprobe_args1
^
/usr/include/bpf/bpf_tracing.h:355:33: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                                       ^~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:54:32: note: expanded from macro 'PT_REGS_PARM1'
#define PT_REGS_PARM1(x) ((x)->di)
                          ~~~  ^
./examples/tcpconnect/tcpconnect.c:113:5: error: no member named 'ax' in 'struct pt_regs'
int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:411:20: note: expanded from macro 'BPF_KRETPROBE'
        return ____##name(___bpf_kretprobe_args(args));                     \
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:395:2: note: expanded from macro '___bpf_kretprobe_args'
        ___bpf_apply(___bpf_kretprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
note: (skipping 1 expansions in backtrace; use -fmacro-backtrace-limit=0 to see all)
<scratch space>:58:1: note: expanded from here
___bpf_kretprobe_args1
^
/usr/include/bpf/bpf_tracing.h:393:36: note: expanded from macro '___bpf_kretprobe_args1'
        ___bpf_kretprobe_args0(), (void *)PT_REGS_RC(ctx)
                                          ^~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:61:29: note: expanded from macro 'PT_REGS_RC'
#define PT_REGS_RC(x) ((x)->ax)
                       ~~~  ^
2 errors generated.
yuval-k commented 2 years ago

Ahh ok this makes sense; a krpobe that uses the arguments needs to be recompiled for arm, as it uses different registers. I'm surprised recompiling didn't work, as it should pick up the right arch. @lgadban , maybe we are not passing the right ARCH defines?

yuval-k commented 2 years ago
nickschuetz commented 2 years ago

Changing the build.sh from

clang-13 -g -O2 -target bpf -D__TARGET_ARCH_x86 -Wall -c $1 -o $2

to

clang-13 -g -O2 -target bpf -D__TARGET_ARCH_ARM -Wall -c $1 -o $2

worked.

$ ./builder/build.sh ./examples/tcpconnect/tcpconnect.c tcpconnect.o
./examples/tcpconnect/tcpconnect.c:107:5: warning: implicit declaration of function 'PT_REGS_PARM1' is invalid in C99 [-Wimplicit-function-declaration]
int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
    ^
/usr/include/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^
/usr/include/bpf/bpf_tracing.h:365:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
/usr/include/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:42:1: note: expanded from here
___bpf_kprobe_args1
^
/usr/include/bpf/bpf_tracing.h:355:33: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                                       ^
./examples/tcpconnect/tcpconnect.c:107:5: warning: cast to 'void *' from smaller integer type 'int' [-Wint-to-void-pointer-cast]
int BPF_KPROBE(tcp_v4_connect, struct sock *sk)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'
        return ____##name(___bpf_kprobe_args(args));                        \
                          ^~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:365:2: note: expanded from macro '___bpf_kprobe_args'
        ___bpf_apply(___bpf_kprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
/usr/include/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:42:1: note: expanded from here
___bpf_kprobe_args1
^
/usr/include/bpf/bpf_tracing.h:355:25: note: expanded from macro '___bpf_kprobe_args1'
        ___bpf_kprobe_args0(), (void *)PT_REGS_PARM1(ctx)
                               ^~~~~~~~~~~~~~~~~~~~~~~~~~
./examples/tcpconnect/tcpconnect.c:113:5: warning: implicit declaration of function 'PT_REGS_RC' is invalid in C99 [-Wimplicit-function-declaration]
int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
    ^
/usr/include/bpf/bpf_tracing.h:411:20: note: expanded from macro 'BPF_KRETPROBE'
        return ____##name(___bpf_kretprobe_args(args));                     \
                          ^
/usr/include/bpf/bpf_tracing.h:395:2: note: expanded from macro '___bpf_kretprobe_args'
        ___bpf_apply(___bpf_kretprobe_args, ___bpf_narg(args))(args)
        ^
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
/usr/include/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:58:1: note: expanded from here
___bpf_kretprobe_args1
^
/usr/include/bpf/bpf_tracing.h:393:36: note: expanded from macro '___bpf_kretprobe_args1'
        ___bpf_kretprobe_args0(), (void *)PT_REGS_RC(ctx)
                                          ^
./examples/tcpconnect/tcpconnect.c:113:5: warning: cast to 'void *' from smaller integer type 'int' [-Wint-to-void-pointer-cast]
int BPF_KRETPROBE(tcp_v4_connect_ret, int ret)
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:411:20: note: expanded from macro 'BPF_KRETPROBE'
        return ____##name(___bpf_kretprobe_args(args));                     \
                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:395:2: note: expanded from macro '___bpf_kretprobe_args'
        ___bpf_apply(___bpf_kretprobe_args, ___bpf_narg(args))(args)
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/bpf/bpf_tracing.h:299:29: note: expanded from macro '___bpf_apply'
#define ___bpf_apply(fn, n) ___bpf_concat(fn, n)
                            ^
/usr/include/bpf/bpf_tracing.h:298:29: note: expanded from macro '___bpf_concat'
#define ___bpf_concat(a, b) a ## b
                            ^
<scratch space>:58:1: note: expanded from here
___bpf_kretprobe_args1
^
/usr/include/bpf/bpf_tracing.h:393:28: note: expanded from macro '___bpf_kretprobe_args1'
        ___bpf_kretprobe_args0(), (void *)PT_REGS_RC(ctx)
                                  ^~~~~~~~~~~~~~~~~~~~~~~
4 warnings generated.

But now when I run it I get:

$ go run -exec sudo ./bee/main.go run tcpconnect.o
 SUCCESS  Fetching program from file: tcpconnect.o
  ERROR   Loading BPF program and maps into Kernel
Error: program tcp_v4_connect: call at 0: reference to missing symbol "PT_REGS_PARM1"
2022/01/10 19:34:00 exiting: program tcp_v4_connect: call at 0: reference to missing symbol "PT_REGS_PARM1"
exit status 1
lgadban commented 2 years ago

Ahh ok this makes sense; a krpobe that uses the arguments needs to be recompiled for arm, as it uses different registers. I'm surprised recompiling didn't work, as it should pick up the right arch. @lgadban , maybe we are not passing the right ARCH defines?

Yeah we are currently hardcoding to x86 -- we definitely need to make the build more generic

yuval-k commented 2 years ago

@nickschuetz try with -D__TARGET_ARCH_arm64. this should work for you. you can see the full list in https://github.com/torvalds/linux/blob/78d8b34751cf3c61b8dcd6ac40b0fc453de3c6a3/tools/lib/bpf/bpf_tracing.h#L6

nickschuetz commented 2 years ago

That did it @yuval-k. A clean build of tcpconnect on arm64 without warnings or errors.

It now runs as expected as well.

lgadban commented 2 years ago

tracking first-class arm64 support here: https://github.com/solo-io/bumblebee/issues/55

sunfuze commented 1 year ago

Also need change vmlinux.h to arm64 version or building raise "error: incomplete definition of type 'struct user_pt_regs'".