advanced03-AF_XDP: Can't setup AF_XDP socket "Operation not permitted"

[Bresenham]

I am using a docker-container with

Linux 4.18.0-25-generic

installed. I can successfully compile your advanced03-AF_XDP example, but the moment I try to deploy it on the target server (Debian 4.19.98-1) via sudo ./af_xdp_user --dev eth20 --force I get:

ibbpf: BPF log buffer:
unknown opcode 66
ERROR: Can't setup AF_XDP socket "Operation not permitted"

The server has all three flags set (verified via /boot/config-4.19.0-8-amd64):

CONFIG_BPF=y CONFIG_BPF_SYSCALL=y CONFIG_XDP_SOCKETS=y

Any ideas?

[tohojo]

Maximilian Gaul notifications@github.com writes:

I am using a docker-container with

Linux 4.18.0-25-generic

installed. I can successfully compile your advanced03-AF_XDP example, but the moment I try to deploy it on the target server (Debian 4.19.98-1) via sudo ./af_xdp_user --dev eth20 --force I get:

ibbpf: BPF log buffer:
unknown opcode 66
ERROR: Can't setup AF_XDP socket "Operation not permitted"

The server has all three flags set:

CONFIG_BPF=y CONFIG_BPF_SYSCALL=y CONFIG_XDP_SOCKETS=y

Any ideas?

You'll need a newer kernel for AF_XDP...

[Bresenham]

Why is that? I often read that this feature was added with Kernel 4.18. Do you know which is the minimal kernel that supports your examples (or AF_XDP in general)? As described here: https://www.kernel.org/doc/html/v4.18/networking/af_xdp.html

[tohojo]

Why is that? I often read that this feature was added with Kernel 4.18.

Parts, but not all.

Do you know which is the minimal kernel that supports your examples (or AF_XDP in general)? As described here: https://www.kernel.org/doc/html/v4.18/networking/af_xdp.html

Well, the newer the better. For the rest of the tutorial, you should be fine with 4.18. For AF_XDP, I'm not sure. 5.1 should be safe, I think, but @bjoto or @magnus-karlsson probably know better...

[Bresenham]

I am doing both compiling and running on my local machine now. I upgraded to 5.3.0-28-generic. First, I created the virtual ethernet interface:

sudo ./testenv.sh setup --name="veth-basic02"

then compiled the example code:

$/xdp-tutorial/advanced03-AF_XDP: make

and then tried to attach the XDP-code to the previously created virtual interface:

sudo ./af_xdp_user --dev veth-basic02 --force

Still produces same output: ERROR: Can't setup AF_XDP socket "Operation not permitted"

I am not too familiar with XDP in general, but do I need a special driver for the (virtual) ethernet interface to function in harmony with AF_XDP?

[tohojo]

Maximilian Gaul notifications@github.com writes:

I upgraded to 5.3.0-28-generic but it still does not work (compiling works fine though) :(

Still the same error message? Which version of LLVM are you using to compile it?

[Bresenham]

$ llvm-config --version -->6.0.0

Edit: Just noticed, that the latest llvm version is 9.0. I don't know why my Kubuntu does not want to upgrade... Edit_2: I just upgraded to llvm-9. If I compile the whole project with llc-9 instead of llc I get:

cc -Wall -I../libbpf/src/build/usr/include/ -g -I../headers/ -L../libbpf/src -o af_xdp_user ../common/common_libbpf.o ../common/common_params.o ../common/common_user_bpf_xdp.o \
 af_xdp_user.c -l:libbpf.a -lelf  -lpthread
cc -Wall -I../libbpf/src/build/usr/include/ -g -I../headers/ -L../libbpf/src -o xdp_stats ../common/common_libbpf.o ../common/common_params.o ../common/common_user_bpf_xdp.o \
 xdp_stats.c -l:libbpf.a -lelf  -lpthread
clang -S \
    -target bpf \
    -D __BPF_TRACING__ \
    -I../libbpf/src/build/usr/include/ -I../headers/ \
    -Wall \
    -Wno-unused-value \
    -Wno-pointer-sign \
    -Wno-compare-distinct-pointer-types \
    -Werror \
    -O2 -emit-llvm -c -g -o af_xdp_kern.ll af_xdp_kern.c
llc-9 -march=bpf -filetype=obj -o af_xdp_kern.o af_xdp_kern.ll
llc-9: error: llc-9: af_xdp_kern.ll:127:200: error: invalid field 'variables'
!54 = distinct !DISubprogram(name: "xdp_sock_prog", scope: !3, file: !3, line: 22, type: !55, isLocal: false, isDefinition: true, scopeLine: 23, flags: DIFlagPrototyped, isOptimized: true, unit: !2, variables: !65)

But I still get a af_xdp_user-binary. But again, same error message if I want to put the program into the interface:

$ sudo ./af_xdp_user --dev veth-basic02 --force ERROR: Can't setup AF_XDP socket "Operation not permitted"

[tohojo]

Maximilian Gaul notifications@github.com writes:

$ llvm-config --version 6.0.0

Hmm, I think that ought to work. Could you please post the output of 'llvm-objdump -S af_xdp_kern.o'?

[tohojo]

Toke Høiland-Jørgensen toke@toke.dk writes:

Maximilian Gaul notifications@github.com writes:

$ llvm-config --version 6.0.0

Hmm, I think that ought to work. Could you please post the output of 'llvm-objdump -S af_xdp_kern.o'?

Also, what's the full output of uname -a on both the host where you compile and the host you are trying to run this on?

[Bresenham]

The Obj-Dump:

$ llvm-objdump -S af_xdp_kern.o 

af_xdp_kern.o:  file format ELF64-BPF

Disassembly of section xdp_sock:
xdp_sock_prog:
; {
       0:       61 11 10 00 00 00 00 00         r1 = *(u32 *)(r1 + 16)
; int index = ctx->rx_queue_index;
       1:       63 1a fc ff 00 00 00 00         *(u32 *)(r10 - 4) = r1
       2:       bf a2 00 00 00 00 00 00         r2 = r10
; int xdp_sock_prog(struct xdp_md *ctx)
       3:       07 02 00 00 fc ff ff ff         r2 += -4
; pkt_count = bpf_map_lookup_elem(&xdp_stats_map, &index);
       4:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00         r1 = 0 ll
       6:       85 00 00 00 01 00 00 00         call 1
; if (pkt_count) {
       7:       15 00 07 00 00 00 00 00         if r0 == 0 goto +7 <LBB0_2>
; if ((*pkt_count)++ & 1)
       8:       61 01 00 00 00 00 00 00         r1 = *(u32 *)(r0 + 0)
       9:       bf 12 00 00 00 00 00 00         r2 = r1
      10:       07 02 00 00 01 00 00 00         r2 += 1
      11:       63 20 00 00 00 00 00 00         *(u32 *)(r0 + 0) = r2
      12:       b7 00 00 00 02 00 00 00         r0 = 2
      13:       57 01 00 00 01 00 00 00         r1 &= 1
      14:       55 01 0d 00 00 00 00 00         if r1 != 0 goto +13 <LBB0_4>

LBB0_2:
      15:       bf a2 00 00 00 00 00 00         r2 = r10
; if (bpf_map_lookup_elem(&xsks_map, &index))
      16:       07 02 00 00 fc ff ff ff         r2 += -4
      17:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00         r1 = 0 ll
      19:       85 00 00 00 01 00 00 00         call 1
      20:       bf 01 00 00 00 00 00 00         r1 = r0
      21:       b7 00 00 00 02 00 00 00         r0 = 2
      22:       15 01 05 00 00 00 00 00         if r1 == 0 goto +5 <LBB0_4>
; return bpf_redirect_map(&xsks_map, index, 0);
      23:       61 a2 fc ff 00 00 00 00         r2 = *(u32 *)(r10 - 4)
      24:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00         r1 = 0 ll
      26:       b7 03 00 00 00 00 00 00         r3 = 0
      27:       85 00 00 00 33 00 00 00         call 51

LBB0_4:
; }
      28:       95 00 00 00 00 00 00 00         exit

And uname:

$ uname -a
Linux lx002385 5.3.0-28-generic #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Btw. as I mentioned previously: to make things easier for me I am now doing everything on the same machine until I get things working correctly

[magnus-karlsson]

Are you using xdpsock and libbpf from the same kernel as you are running? Sorry, do not have the conversation history so you might have answered this already.

/Magnus

On Thu, Feb 20, 2020 at 1:48 PM Maximilian Gaul notifications@github.com wrote:

The Obj-Dump:

$ llvm-objdump -S af_xdp_kern.o

af_xdp_kern.o: file format ELF64-BPF

Disassembly of section xdp_sock: xdp_sock_prog: ; { 0: 61 11 10 00 00 00 00 00 r1 = (u32 )(r1 + 16) ; int index = ctx->rx_queue_index; 1: 63 1a fc ff 00 00 00 00 (u32 )(r10 - 4) = r1 2: bf a2 00 00 00 00 00 00 r2 = r10 ; int xdp_sock_prog(struct xdp_md ctx) 3: 07 02 00 00 fc ff ff ff r2 += -4 ; pkt_count = bpf_map_lookup_elem(&xdp_stats_map, &index); 4: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll 6: 85 00 00 00 01 00 00 00 call 1 ; if (pkt_count) { 7: 15 00 07 00 00 00 00 00 if r0 == 0 goto +7 ; if ((pkt_count)++ & 1) 8: 61 01 00 00 00 00 00 00 r1 = (u32 )(r0 + 0) 9: bf 12 00 00 00 00 00 00 r2 = r1 10: 07 02 00 00 01 00 00 00 r2 += 1 11: 63 20 00 00 00 00 00 00 (u32 )(r0 + 0) = r2 12: b7 00 00 00 02 00 00 00 r0 = 2 13: 57 01 00 00 01 00 00 00 r1 &= 1 14: 55 01 0d 00 00 00 00 00 if r1 != 0 goto +13

LBB0_2: 15: bf a2 00 00 00 00 00 00 r2 = r10 ; if (bpf_map_lookup_elem(&xsks_map, &index)) 16: 07 02 00 00 fc ff ff ff r2 += -4 17: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll 19: 85 00 00 00 01 00 00 00 call 1 20: bf 01 00 00 00 00 00 00 r1 = r0 21: b7 00 00 00 02 00 00 00 r0 = 2 22: 15 01 05 00 00 00 00 00 if r1 == 0 goto +5 ; return bpf_redirect_map(&xsks_map, index, 0); 23: 61 a2 fc ff 00 00 00 00 r2 = (u32 )(r10 - 4) 24: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll 26: b7 03 00 00 00 00 00 00 r3 = 0 27: 85 00 00 00 33 00 00 00 call 51

LBB0_4: ; } 28: 95 00 00 00 00 00 00 00 exit

And uname:

$ uname -a Linux lx002385 5.3.0-28-generic #30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Btw. as I mentioned previously: to make things easier for me I am now doing everything on the same machine until I get things working correctly

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/xdp-project/xdp-tutorial/issues/100?email_source=notifications&email_token=AASGUELC6LD22LTOCCPDTDTRDZ3YRA5CNFSM4KYJ5MV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMNX5WI#issuecomment-589004505, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASGUEJFMOBMULP5RLLXNOLRDZ3YRANCNFSM4KYJ5MVQ .

[Bresenham]

Regarding libbpf, I am using the one your are supplying as a submodule with your repository. In fact, I tried to use the libbpf from the original repository (this means I just replaced the contents of libbps-folder with the contents of the libbpf-repository) but compilation failed so I reversed the changes.

I don't really know what xdpsock is..I am just using your repository. Didn't change a lot.

[tohojo]

Maximilian Gaul notifications@github.com writes:

The Obj-Dump:

$ llvm-objdump -S af_xdp_kern.o 

af_xdp_kern.o:  file format ELF64-BPF

Disassembly of section xdp_sock:
xdp_sock_prog:
; {
       0:       61 11 10 00 00 00 00 00         r1 = *(u32 *)(r1 + 16)
; int index = ctx->rx_queue_index;
       1:       63 1a fc ff 00 00 00 00         *(u32 *)(r10 - 4) = r1
       2:       bf a2 00 00 00 00 00 00         r2 = r10
; int xdp_sock_prog(struct xdp_md *ctx)
       3:       07 02 00 00 fc ff ff ff         r2 += -4
; pkt_count = bpf_map_lookup_elem(&xdp_stats_map, &index);
       4:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00         r1 = 0 ll
       6:       85 00 00 00 01 00 00 00         call 1
; if (pkt_count) {
       7:       15 00 07 00 00 00 00 00         if r0 == 0 goto +7 <LBB0_2>
; if ((*pkt_count)++ & 1)
       8:       61 01 00 00 00 00 00 00         r1 = *(u32 *)(r0 + 0)
       9:       bf 12 00 00 00 00 00 00         r2 = r1
      10:       07 02 00 00 01 00 00 00         r2 += 1
      11:       63 20 00 00 00 00 00 00         *(u32 *)(r0 + 0) = r2
      12:       b7 00 00 00 02 00 00 00         r0 = 2
      13:       57 01 00 00 01 00 00 00         r1 &= 1
      14:       55 01 0d 00 00 00 00 00         if r1 != 0 goto +13 <LBB0_4>

LBB0_2:
      15:       bf a2 00 00 00 00 00 00         r2 = r10
; if (bpf_map_lookup_elem(&xsks_map, &index))
      16:       07 02 00 00 fc ff ff ff         r2 += -4
      17:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00         r1 = 0 ll
      19:       85 00 00 00 01 00 00 00         call 1
      20:       bf 01 00 00 00 00 00 00         r1 = r0
      21:       b7 00 00 00 02 00 00 00         r0 = 2
      22:       15 01 05 00 00 00 00 00         if r1 == 0 goto +5 <LBB0_4>
; return bpf_redirect_map(&xsks_map, index, 0);
      23:       61 a2 fc ff 00 00 00 00         r2 = *(u32 *)(r10 - 4)
      24:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00         r1 = 0 ll
      26:       b7 03 00 00 00 00 00 00         r3 = 0
      27:       85 00 00 00 33 00 00 00         call 51

LBB0_4:
; }
      28:       95 00 00 00 00 00 00 00         exit

That's the same output I get. Can you load the XDP program itself? Try:

sudo ip link set dev xdp object af_xdp_kern.o section xdp_sock

[Bresenham]

$ sudo ip link set dev veth-basic02 xdp object af_xdp_kern.o section xdp_sock

Note: 16 bytes struct bpf_elf_map fixup performed due to size mismatch!
Map object 'xsks_map' rejected: Operation not permitted (1)!
 - Type:         17
 - Identifier:   0
 - Pinning:      0
 - Size key:     4
 - Size value:   4
 - Max elems:    64
 - Flags:        0

Error loading maps into kernel!
Error fetching ELF ancillary data!

Edit: I am wondering why you recommend to use a custom "C-code based" loader in the tutorial to load the XDP-Program into the kernel while there is a much simpler version with the ip link-statement you just mentioned.

[chaudron]

Take a look at this patch, waiting to be merged ;) Maybe it's related

https://github.com/chaudron/xdp-tutorial/commit/3f89b0e90f67eabd132bcecda9253d2f08144c17

[Bresenham]

I tried your patch even though it does not affect me because I am using kernel 5.3, result: Doesn't change anything

Edit: This is the solution: https://stackoverflow.com/questions/60322147/xdp-program-ip-link-error-prog-section-rejected-operation-not-permitted