Attach another XDP program to an already-attached list of programs?

[stevelorenz]

Dear XDP developers,

Thanks so much for your work. May I ask about the possibility to try attaching/appending another XDP program to an already-attached list of programs or replacing an attached XDP program? I am currently developing a XDP-based traffic monitoring algorithm (as a PhD student :wink: ). I wrote the XDP program but face the problem when I want to integrate it with OVS's AF_XDP datapath and DPDK's AF_XDP PMD. Both of them use the static int xsk_setup_xdp_prog(struct xsk_socket *xsk) function (provided by libbpf) to setup the AF_XDP socket and there is a "dummy" built-in XDP program be loaded inside this function (by calling xsk_load_xdp_prog() function). Then I can not attach my monitor XDP program easily on the same veth interface. This limitation makes it difficult to test my approach using OVS AF_XDP directly. I did some workarounds (write my own simple "switch" using XDP) but it would be great if the additional XDP program can be attached. I see in the README of libxdp that:

It is expected that this restriction will be lifted in an upcoming kernel version, after which xdp_program__attach() will simply add the program being loaded into the existing chain of programs running on the interface.

Is it possible to use this feature in the kernel version 5.8? Or I can test this feature by compiling the kernel source from a special testing repo? Thanks so much for your patience and answer.

With best regards,

Zuo Xiang

[tohojo]

Not yet, unfortunately; the required support is still missing from the kernel. I just got back from vacation this week, and finally getting this fixed is near the top of my list. I'll update this issue when this lands in the kernel.

Note, however, that if you just want to monitor the traffic, you can probably just use the same mechanism that xdpdump uses to attach to an existing XDP program. This should work on a 5.8 kernel - you can just try running xdpdump while that other program is attached and see if it works; then look at the xdpdump source for how it does this :)

[stevelorenz]

Thanks so much for your helpful response :+1: . I will check the xdpdump source. I will put you (as the XDP expert) in the acknowledgement of my thesis :stuck_out_tongue_winking_eye: . Best wishes for your todo list and look forward to the new kernel.

[tohojo]

Zuo Xiang notifications@github.com writes:

Thanks so much for your helpful response :+1: . I will check the xdpdump source. I will put you (as the XDP expert) in the acknowledgement of my thesis :stuck_out_tongue_winking_eye: . Best wishes for your todo list and look forward to the new kernel.

You're welcome - and thanks! Good luck with your thesis!

[stevelorenz]

Zuo Xiang notifications@github.com writes: Thanks so much for your helpful response +1 . I will check the xdpdump source. I will put you (as the XDP expert) in the acknowledgement of my thesis stuck_out_tongue_winking_eye . Best wishes for your todo list and look forward to the new kernel. You're welcome - and thanks! Good luck with your thesis!

May I get some enlightening tips from the expert for one issue when I updated xdp-tools from the v0.0.3 to v1.0.0? All programs get compiled without any issues :+1:. Unfortunately when I try to use the xdp-loader to load the program, I get the following error about "Argument list too long". I have tried ulimit -s unlimited and it still failed.

Current rlimit 67108864 already >= minimum 1048576
Loading 1 files on interface 'eth0'.
XDP program 0: Run prio: 50. Chain call actions: XDP_PASS
libxdp: Failed to load program xdp_prog_pass: Argument list too long
Couldn't attach XDP program on iface 'eth0': Argument list too long(-7)

• Kernel version: 5.4.0-42
• Distribution: Ubuntu 20.04

Thanks so much for your help.

[tohojo]

Hmm, not sure why you'd get that error exactly (could you try running with '-vv' to get some more debug output)?

However, there have been some issues with extension programs not being equivalent with the main programs which may be what you're hitting; that has only just been fixed in the kernel in the latest bpf-next tree, so if that is what you are hitting, you'll have to upgrade your kernel I'm afraid...

[stevelorenz]

Hmm, not sure why you'd get that error exactly (could you try running with '-vv' to get some more debug output)? However, there have been some issues with extension programs not being equivalent with the main programs which may be what you're hitting; that has only just been fixed in the kernel in the latest bpf-next tree, so if that is what you are hitting, you'll have to upgrade your kernel I'm afraid...

The output of -vv:

Click to expand!

``` Current rlimit 67108864 already >= minimum 1048576 Loading 1 files on interface 'eth0'. libbpf: loading ./xdp_pass_kern.o libbpf: section(1) .strtab, size 162, link 0, flags 0, type=3 libbpf: skip section(1) .strtab libbpf: section(2) .text, size 0, link 0, flags 6, type=1 libbpf: skip section(2) .text libbpf: section(3) xdp, size 16, link 0, flags 6, type=1 libbpf: found program xdp libbpf: section(4) license, size 4, link 0, flags 3, type=1 libbpf: license of ./xdp_pass_kern.o is GPL libbpf: section(5) .debug_str, size 300, link 0, flags 30, type=1 libbpf: skip section(5) .debug_str libbpf: section(6) .debug_abbrev, size 186, link 0, flags 0, type=1 libbpf: skip section(6) .debug_abbrev libbpf: section(7) .debug_info, size 289, link 0, flags 0, type=1 libbpf: skip section(7) .debug_info libbpf: section(8) .rel.debug_info, size 464, link 17, flags 0, type=9 libbpf: skip relo .rel.debug_info(8) for section(7) libbpf: section(9) .BTF, size 531, link 0, flags 0, type=1 libbpf: section(10) .rel.BTF, size 16, link 17, flags 0, type=9 libbpf: skip relo .rel.BTF(10) for section(9) libbpf: section(11) .BTF.ext, size 80, link 0, flags 0, type=1 libbpf: section(12) .rel.BTF.ext, size 32, link 17, flags 0, type=9 libbpf: skip relo .rel.BTF.ext(12) for section(11) libbpf: section(13) .debug_frame, size 40, link 0, flags 0, type=1 libbpf: skip section(13) .debug_frame libbpf: section(14) .rel.debug_frame, size 32, link 17, flags 0, type=9 libbpf: skip relo .rel.debug_frame(14) for section(13) libbpf: section(15) .debug_line, size 154, link 0, flags 0, type=1 libbpf: skip section(15) .debug_line libbpf: section(16) .rel.debug_line, size 16, link 17, flags 0, type=9 libbpf: skip relo .rel.debug_line(16) for section(15) libbpf: section(17) .symtab, size 768, link 1, flags 0, type=2 libbpf: looking for externs among 32 symbols... libbpf: collected 0 externs total libbpf: loading ./xdp_pass_kern.o libbpf: section(1) .strtab, size 162, link 0, flags 0, type=3 libbpf: skip section(1) .strtab libbpf: section(2) .text, size 0, link 0, flags 6, type=1 libbpf: skip section(2) .text libbpf: section(3) xdp, size 16, link 0, flags 6, type=1 libbpf: found program xdp libbpf: section(4) license, size 4, link 0, flags 3, type=1 libbpf: license of ./xdp_pass_kern.o is GPL libbpf: section(5) .debug_str, size 300, link 0, flags 30, type=1 libbpf: skip section(5) .debug_str libbpf: section(6) .debug_abbrev, size 186, link 0, flags 0, type=1 libbpf: skip section(6) .debug_abbrev libbpf: section(7) .debug_info, size 289, link 0, flags 0, type=1 libbpf: skip section(7) .debug_info libbpf: section(8) .rel.debug_info, size 464, link 17, flags 0, type=9 libbpf: skip relo .rel.debug_info(8) for section(7) libbpf: section(9) .BTF, size 531, link 0, flags 0, type=1 libbpf: section(10) .rel.BTF, size 16, link 17, flags 0, type=9 libbpf: skip relo .rel.BTF(10) for section(9) libbpf: section(11) .BTF.ext, size 80, link 0, flags 0, type=1 libbpf: section(12) .rel.BTF.ext, size 32, link 17, flags 0, type=9 libbpf: skip relo .rel.BTF.ext(12) for section(11) libbpf: section(13) .debug_frame, size 40, link 0, flags 0, type=1 libbpf: skip section(13) .debug_frame libbpf: section(14) .rel.debug_frame, size 32, link 17, flags 0, type=9 libbpf: skip relo .rel.debug_frame(14) for section(13) libbpf: section(15) .debug_line, size 154, link 0, flags 0, type=1 libbpf: skip section(15) .debug_line libbpf: section(16) .rel.debug_line, size 16, link 17, flags 0, type=9 libbpf: skip relo .rel.debug_line(16) for section(15) libbpf: section(17) .symtab, size 768, link 1, flags 0, type=2 libbpf: looking for externs among 32 symbols... libbpf: collected 0 externs total libxdp: DATASEC '.xdp_run_config' not found. XDP program 0: Run prio: 50. Chain call actions: XDP_PASS libxdp: Generating multi-prog dispatcher for 1 programs libxdp: Looking for './xdp-dispatcher.o' libxdp: Loading XDP program from './xdp-dispatcher.o' section 'xdp/dispatcher' libbpf: loading ./xdp-dispatcher.o libbpf: section(1) .strtab, size 405, link 0, flags 0, type=3 libbpf: skip section(1) .strtab libbpf: section(2) .text, size 480, link 0, flags 6, type=1 libbpf: found program .text libbpf: section(3) xdp/dispatcher, size 1064, link 0, flags 6, type=1 libbpf: found program xdp/dispatcher libbpf: section(4) .relxdp/dispatcher, size 320, link 24, flags 0, type=9 libbpf: section(5) .rodata, size 84, link 0, flags 2, type=1 libbpf: section(6) license, size 4, link 0, flags 3, type=1 libbpf: license of ./xdp-dispatcher.o is GPL libbpf: section(7) xdp_metadata, size 8, link 0, flags 3, type=1 libbpf: skip section(7) xdp_metadata libbpf: section(8) .debug_str, size 468, link 0, flags 30, type=1 libbpf: skip section(8) .debug_str libbpf: section(9) .debug_loc, size 343, link 0, flags 0, type=1 libbpf: skip section(9) .debug_loc libbpf: section(10) .rel.debug_loc, size 48, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_loc(10) for section(9) libbpf: section(11) .debug_abbrev, size 312, link 0, flags 0, type=1 libbpf: skip section(11) .debug_abbrev libbpf: section(12) .debug_info, size 1139, link 0, flags 0, type=1 libbpf: skip section(12) .debug_info libbpf: section(13) .rel.debug_info, size 1520, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_info(13) for section(12) libbpf: section(14) .debug_ranges, size 48, link 0, flags 0, type=1 libbpf: skip section(14) .debug_ranges libbpf: section(15) .rel.debug_ranges, size 64, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_ranges(15) for section(14) libbpf: section(16) .BTF, size 2958, link 0, flags 0, type=1 libbpf: section(17) .rel.BTF, size 48, link 24, flags 0, type=9 libbpf: skip relo .rel.BTF(17) for section(16) libbpf: section(18) .BTF.ext, size 1968, link 0, flags 0, type=1 libbpf: section(19) .rel.BTF.ext, size 1984, link 24, flags 0, type=9 libbpf: skip relo .rel.BTF.ext(19) for section(18) libbpf: section(20) .debug_frame, size 280, link 0, flags 0, type=1 libbpf: skip section(20) .debug_frame libbpf: section(21) .rel.debug_frame, size 352, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_frame(21) for section(20) libbpf: section(22) .debug_line, size 612, link 0, flags 0, type=1 libbpf: skip section(22) .debug_line libbpf: section(23) .rel.debug_line, size 32, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_line(23) for section(22) libbpf: section(24) .symtab, size 1920, link 1, flags 0, type=2 libbpf: looking for externs among 80 symbols... libbpf: collected 0 externs total libbpf: map 'xdp_disp.rodata' (global data): at sec_idx 5, offset 0, flags 480. libbpf: map 0 is "xdp_disp.rodata" libbpf: collecting relocating info for: 'xdp/dispatcher' libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 1 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 1 libbpf: relo for shdr 2, symb 69, value 0, type 2, bind 1, name 390 ('prog0'), insn 6 libbpf: relo for shdr 2, symb 70, value 48, type 2, bind 1, name 375 ('prog1'), insn 17 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 18 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 18 libbpf: relo for shdr 2, symb 71, value 96, type 2, bind 1, name 299 ('prog2'), insn 30 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 31 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 31 libbpf: relo for shdr 2, symb 72, value 144, type 2, bind 1, name 293 ('prog3'), insn 43 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 44 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 44 libbpf: relo for shdr 2, symb 73, value 192, type 2, bind 1, name 287 ('prog4'), insn 56 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 57 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 57 libbpf: relo for shdr 2, symb 74, value 240, type 2, bind 1, name 281 ('prog5'), insn 69 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 70 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 70 libbpf: relo for shdr 2, symb 75, value 288, type 2, bind 1, name 275 ('prog6'), insn 82 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 83 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 83 libbpf: relo for shdr 2, symb 76, value 336, type 2, bind 1, name 269 ('prog7'), insn 95 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 96 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 96 libbpf: relo for shdr 2, symb 77, value 384, type 2, bind 1, name 263 ('prog8'), insn 108 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 109 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 109 libbpf: relo for shdr 2, symb 78, value 432, type 2, bind 1, name 257 ('prog9'), insn 121 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 122 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 122 libbpf: loading ./xdp-dispatcher.o libbpf: section(1) .strtab, size 405, link 0, flags 0, type=3 libbpf: skip section(1) .strtab libbpf: section(2) .text, size 480, link 0, flags 6, type=1 libbpf: found program .text libbpf: section(3) xdp/dispatcher, size 1064, link 0, flags 6, type=1 libbpf: found program xdp/dispatcher libbpf: section(4) .relxdp/dispatcher, size 320, link 24, flags 0, type=9 libbpf: section(5) .rodata, size 84, link 0, flags 2, type=1 libbpf: section(6) license, size 4, link 0, flags 3, type=1 libbpf: license of ./xdp-dispatcher.o is GPL libbpf: section(7) xdp_metadata, size 8, link 0, flags 3, type=1 libbpf: skip section(7) xdp_metadata libbpf: section(8) .debug_str, size 468, link 0, flags 30, type=1 libbpf: skip section(8) .debug_str libbpf: section(9) .debug_loc, size 343, link 0, flags 0, type=1 libbpf: skip section(9) .debug_loc libbpf: section(10) .rel.debug_loc, size 48, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_loc(10) for section(9) libbpf: section(11) .debug_abbrev, size 312, link 0, flags 0, type=1 libbpf: skip section(11) .debug_abbrev libbpf: section(12) .debug_info, size 1139, link 0, flags 0, type=1 libbpf: skip section(12) .debug_info libbpf: section(13) .rel.debug_info, size 1520, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_info(13) for section(12) libbpf: section(14) .debug_ranges, size 48, link 0, flags 0, type=1 libbpf: skip section(14) .debug_ranges libbpf: section(15) .rel.debug_ranges, size 64, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_ranges(15) for section(14) libbpf: section(16) .BTF, size 2958, link 0, flags 0, type=1 libbpf: section(17) .rel.BTF, size 48, link 24, flags 0, type=9 libbpf: skip relo .rel.BTF(17) for section(16) libbpf: section(18) .BTF.ext, size 1968, link 0, flags 0, type=1 libbpf: section(19) .rel.BTF.ext, size 1984, link 24, flags 0, type=9 libbpf: skip relo .rel.BTF.ext(19) for section(18) libbpf: section(20) .debug_frame, size 280, link 0, flags 0, type=1 libbpf: skip section(20) .debug_frame libbpf: section(21) .rel.debug_frame, size 352, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_frame(21) for section(20) libbpf: section(22) .debug_line, size 612, link 0, flags 0, type=1 libbpf: skip section(22) .debug_line libbpf: section(23) .rel.debug_line, size 32, link 24, flags 0, type=9 libbpf: skip relo .rel.debug_line(23) for section(22) libbpf: section(24) .symtab, size 1920, link 1, flags 0, type=2 libbpf: looking for externs among 80 symbols... libbpf: collected 0 externs total libbpf: map 'xdp_disp.rodata' (global data): at sec_idx 5, offset 0, flags 480. libbpf: map 0 is "xdp_disp.rodata" libbpf: collecting relocating info for: 'xdp/dispatcher' libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 1 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 1 libbpf: relo for shdr 2, symb 69, value 0, type 2, bind 1, name 390 ('prog0'), insn 6 libbpf: relo for shdr 2, symb 70, value 48, type 2, bind 1, name 375 ('prog1'), insn 17 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 18 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 18 libbpf: relo for shdr 2, symb 71, value 96, type 2, bind 1, name 299 ('prog2'), insn 30 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 31 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 31 libbpf: relo for shdr 2, symb 72, value 144, type 2, bind 1, name 293 ('prog3'), insn 43 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 44 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 44 libbpf: relo for shdr 2, symb 73, value 192, type 2, bind 1, name 287 ('prog4'), insn 56 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 57 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 57 libbpf: relo for shdr 2, symb 74, value 240, type 2, bind 1, name 281 ('prog5'), insn 69 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 70 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 70 libbpf: relo for shdr 2, symb 75, value 288, type 2, bind 1, name 275 ('prog6'), insn 82 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 83 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 83 libbpf: relo for shdr 2, symb 76, value 336, type 2, bind 1, name 269 ('prog7'), insn 95 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 96 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 96 libbpf: relo for shdr 2, symb 77, value 384, type 2, bind 1, name 263 ('prog8'), insn 108 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 109 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 109 libbpf: relo for shdr 2, symb 78, value 432, type 2, bind 1, name 257 ('prog9'), insn 121 libbpf: relo for shdr 5, symb 61, value 0, type 3, bind 0, name 0 (''), insn 122 libbpf: found data map 0 (xdp_disp.rodata, sec 5, off 0) for insn 122 libbpf: map 'xdp_disp.rodata': created successfully, fd=5 libbpf: added 60 insn from .text to prog xdp/dispatcher libxdp: DATASEC '.xdp_run_config' not found. libxdp: Verified XDP dispatcher version 1 <= 1 libxdp: Loading multiprog dispatcher for 1 programs libbpf: map 'xdp_disp.rodata': created successfully, fd=5 libbpf: added 60 insn from .text to prog xdp/dispatcher libxdp: Loaded XDP program xdp_dispatcher, got fd 6 libxdp: Linking prog xdp_prog_pass as multiprog entry 0 libbpf: load bpf program failed: Argument list too long libbpf: failed to load program 'xdp' libbpf: failed to load object './xdp_pass_kern.o' libxdp: Failed to load program xdp_prog_pass: Argument list too long Couldn't attach XDP program on iface 'eth0': Argument list too long(-7) ```

I grepped and assume the error happens in xdp_multiprog__link_prog function in libxdp.c. Maybe this requires the update to the latest bpf-next tree. I'll update the kernel (if required) after getting all evalution plots for a paper :see_no_evil:. Thanks so much for your help :+1: .

[tohojo]

Yeah, this sounds like you just need an older kernel. I'll update the error message to be more helpful. Eventually I also want to add backwards compatibility for older kernels, but haven't gotten around to that yet...

[tohojo]

As of commit 4e73b130273529c9f56c9a4837e3b28c40125ca9, it is now possible to add additional XDP programs to an existing one, as long as you're a 5.10 kernel, and everything was loaded with libxdp. We should also be handling fallback on older kernels better now.

So closing this issue - please reopen if you still experience problems.

[stevelorenz]

As of commit 4e73b13, it is now possible to add additional XDP programs to an existing one, as long as you're a 5.10 kernel, and everything was loaded with libxdp. We should also be handling fallback on older kernels better now.

So closing this issue - please reopen if you still experience problems.

Thanks so much for your reply :+1:. I'll update my kernel to 5.10 for tests soon. Have a nice day !