"Newer" libbpf

[tjcw]

I notice that the xdp-tutorial is set up with the "older" libbpf, and I want to build eBPF programs based on the tutorial with the "newer" libbpf. Has anyone got a version of the tutorial which uses the "newer" libbpf ? I am working through issues arising, and currently at a compile error

tjcw@r28b29-n10:~/workspace/xdp-tutorial/common$ make
gcc -g -Wall -I../../libbpf/src//build/usr/include/  -I../headers -c -o common_user_bpf_xdp.o common_user_bpf_xdp.c
In file included from common_user_bpf_xdp.c:1:
../../libbpf/src//build/usr/include/bpf/libbpf.h:70:54: warning: 'enum bpf_link_type' declared inside parameter list will not be visible outside of this definition or declaration
   70 | LIBBPF_API const char *libbpf_bpf_link_type_str(enum bpf_link_type t);
      |                                                      ^~~~~~~~~~~~~

(as you can tell I am not very far through the port). All help with this apprecaited.

[lancewolves]

I don't have a version using the new libbpf v1.0, but in case you did not find it yet, there is a migration guide covering most of the API changes here

[tjcw]

I have my kernel and user code running now; test case is here https://github.com/tjcw/xdp-tutorial/tree/master/ebpf-filter . This is modelled after advanced03-AF_XDP . However I do not succeed in getting the packets delivered to user space. I think the problem is with this code fragment

    if (bpf_map_lookup_elem(&xsks_map, &index))

I cannot see where the xsks_map gets associated with the AF_XDP socket in the user code; in advanced03-AF_XDP I see

        map = bpf_object__find_map_by_name(bpf_obj, "xsks_map");
        xsks_map_fd = bpf_map__fd(map);

but I don't see any use made of xsks_map_fd.

I have run advanced03-AF_XDP with the prototype BPF API, and it works. But when I run my test case with the BPF 1.0 API, it doesn't work. Can someone tell me what I am doing wrong or point me at some code which works ?

[tohojo]

There are a couple of AF_XDP examples in the bpf-examples repository: https://github.com/xdp-project/bpf-examples

That repository also contains a bunch of code using the newer libbpf API, so it's a good place to look for examples until someone gets around to updating this tutorial...

[tjcw]

It looks like my problem is that I need an XDP dispatcher program; and then I need my XDP program to be loaded so that the dispatcher can call it. I will look in the bpf-examples directory to see if I can find how to do that.

[tohojo]

Chris Ward @.***> writes:

It looks like my problem is that I need an XDP dispatcher program; and then I need my XDP program to be loaded so that the dispatcher can call it. I will look in the bpf-examples directory to see if I can find how to do that.

If you create the socket using the libxdp XSK APIs that should take care of loading the XDP dispatcher program for you (unless you explicitly tell it not to)...

[tjcw]

I have converted to using the libxdp APIs, and removed the flag which disables program load. Now I am getting

root@r28b29-n11:/home/tjcw/workspace/xdp-tutorial/ebpf-filter# ./af_xdp_user -S -d eth0 --filename ./af_xdp_kern.o
libbpf: elf: skipping unrecognized data section(7) license2
libxdp: Found func xdp_sock_prog_0 matching xdp_sock_prog_0
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found license2
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: Generating multi-prog dispatcher for 1 programs
libxdp: Looking for './xdp-dispatcher.o'
libxdp: Looking for '/usr/local/lib/bpf/xdp-dispatcher.o'
libxdp: Loading XDP program from '/usr/local/lib/bpf/xdp-dispatcher.o' section 'xdp_dispatcher'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: xdp_metadata_section=xdp_metadata
libxdp: btf_get_datasec looking for xdp_metadata found .rodata
libxdp: btf_get_datasec looking for xdp_metadata found license
libxdp: btf_get_datasec looking for xdp_metadata found xdp_metadata
libxdp: Verified XDP dispatcher version 1 <= 1
libxdp: Loading multiprog dispatcher for 1 programs
libxdp: Loaded XDP program xdp_dispatcher, got fd 5
libxdp: Checking dispatcher compatibility
libxdp: Looking for './xdp-dispatcher.o'
libxdp: Looking for '/usr/local/lib/bpf/xdp-dispatcher.o'
libxdp: Loading XDP program from '/usr/local/lib/bpf/xdp-dispatcher.o' section 'xdp_pass'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libbpf: prog 'xdp_pass': BPF program load failed: Argument list too long
libbpf: prog 'xdp_pass': failed to load: -7
libbpf: failed to load object '/usr/local/lib/bpf/xdp-dispatcher.o'
libxdp: Failed to load program xdp_pass: Argument list too long
libxdp: Compatibility check for dispatcher program failed: Argument list too long
libxdp: Falling back to loading single prog without dispatcher
libxdp: Loaded XDP program xdp_sock_prog_0, got fd 6
libxdp: Replacing XDP fd -1 with 7 on ifindex 2
libxdp: xdp_metadata_section=xdp_metadata
libxdp: btf_get_datasec looking for xdp_metadata found .maps
libxdp: btf_get_datasec looking for xdp_metadata found license
libxdp: btf_get_datasec looking for xdp_metadata found license2
libxdp: DATASEC 'xdp_metadata' not found.
libxdp: Found func xdp_sock_prog_0 matching xdp_sock_prog_0
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found license2
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: Found legacy program with id 156 and 0 component progs
libxdp: xdp_metadata_section=xdp_metadata
libxdp: btf_get_datasec looking for xdp_metadata found .maps
libxdp: btf_get_datasec looking for xdp_metadata found license
libxdp: btf_get_datasec looking for xdp_metadata found license2
libxdp: DATASEC 'xdp_metadata' not found.
xsk_socket__create returns -2
ERROR: Can't setup AF_XDP socket 0 "No such file or directory"
root@r28b29-n11:/home/tjcw/workspace/xdp-tutorial/ebpf-filter# 

(I have added a few debug statements to libxdp but I have not changed the program logic). So it looks like the attempt to load the dispatcher is getting a 'too many arguments' from loading xdp_pass. xdp_pass only has one argument. And after that it tries loading my program without a dispatcher, which fails because the libxdp expects an XDP_METADATA_SECTION in the program. I will look more closely at the bpf-examples to see if I have missed something.

[tjcw]

I have another machine which runs Ubuntu 22.04, and on this machine I can run my test case https://github.com/tjcw/xdp-tutorial/tree/master/ebpf-filter . I can also run the AF_XDP-example from bpf-examples; bpf-examples needs clang 11, so does not build on Ubuntu 20.04. I do not yet know whether my ebpf-filter test runs correctly; the machine is on a somewhat-isolated network.

Part of the xdp-tutorial involves running the xdp_stats utility to display statistics which the eBPF program collects and puts in an eBPF map. However I cannot see the map in the file system ; I was expecting it under /sys/fs/bpf . Have maps moved ?

[tjcw]

My ebpf-filter is not working correctly. It looks like the device isn't being put into the xsks_map; and so

    if (bpf_map_lookup_elem(&xsks_map, &index))

is indicating that the packet should not be intercepted. How do I debug this ?

[tjcw]

I am now seeing bpf_map_lookup_elem(&xsks_map, &index) returning the expected result, but my eBPF program is behaving strangely. As coded, it should parse the packet, drop TCP packets, and pass all others to user space. But what it seems to be doing is passing TCP packets to the network stack and passing ICMP packets to user space; not sure what it is doing with other protocols. It is proving difficult to debug eBPF programs because you cannot use gdb or similar, and you cannot add 'print' statements. Is there a solution to this ?

[tohojo]

There's a bpf_printk() helper you can call and read the output from the trace pipe at /sys/kernel/tracing/trace_pipe

[tjcw]

Thanks, @tohojo . With tracing, I find that some of my problem is that I was not steering traffic to a particular queue.But when I try to do this, I get

root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter# ethtool -N enp25s0 flow-type udp4 action 0
rxclass: Cannot get RX class rule count: Operation not supported
Cannot insert classification rule
root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter#
Should I be using ethtool, and if so what is the right invocation ?

When I try ethtool on a 'dumb' ethernet card in the same machine (enp25s0 is a Pensando SmartNIC) I get

root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter# ethtool -N enp51s0f0 flow-type udp4 action 0 
rmgr: Cannot insert RX class rule: Invalid argument
root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter#

which again does not work, but gives a different message.

[tohojo]

Well, that sounds like your hardware doesn't support that feature... these knobs are hardware-specific. so no idea what the right way of doing this is, really...

[tjcw]

Exploring some more, I have now set up my eBPF program like

SEC("xdp")
int xdp_sock_prog_0(struct xdp_md *ctx)
{

    int index = ctx->rx_queue_index;
    bpf_printk("index=%d returning XDP_PASS\n") ;
    return XDP_PASS;
}

i.e. trace everything and pass everything. However I find that 'ping' is not going through this interface when the test case is running, no trace lines are output, and packets are being received by the userspace application. So I am confused as to what is going on. I wondered if a previous eBPF program is being left running, but 'ip a' says there isn't one after my test case finishes; this is with Ubuntu 22.04, where I was prevoiusly running with Ubuntu 20.04 and had to explicitly unload the eBPF program there.

[tjcw]

I am suspicious as to whether my eBPF program is being loaded. So I instrumented the xdp_programopen_file and xdp_programattach calls; it seems that my program is being loaded, but the behaviour is as if some other eBPF porgram is running. Is there a way of getting independent confirmation that my program is actually being loaded ?

[tohojo]

Chris Ward @.***> writes:

I am suspicious as to whether my eBPF program is being loaded. So I instrumented the xdp_programopen_file and xdp_programattach calls; it seems that my program is being loaded, but the behaviour is as if some other eBPF porgram is running. Is there a way of getting independent confirmation that my program is actually being loaded ?

xdp-loader status should give you a complete list of which program(s) are running on an interface...

[tjcw]

xdp-loader status indicated that old eBPF programs were being left, so I ran ip link set dev enp25s0 xdpgeneric off which cleaned them up. I also added metadata

struct {
        __uint(priority, 1);
        __uint(XDP_PASS, 1);
} XDP_RUN_CONFIG(xdp_sock_prog);

xdp-loader status now shows what I expect, but I am still getting packets sent to user-space even though my eBPF program is like

SEC("xdp")
int xdp_sock_prog(struct xdp_md *ctx)
{

    int index = ctx->rx_queue_index;
    bpf_printk("index=%d returning XDP_PASS\n", index) ;
    return XDP_PASS;
}

I am getting bpf_trace_printk lines like

          <idle>-0       [091] dNs.1  5321.706652: bpf_trace_printk: index=0 returning XDP_PASS

          <idle>-0       [091] dNs.1  5322.730620: bpf_trace_printk: index=0 returning XDP_PASS

What is sending the packets to user space ? Also I still don't know how to access an eBPF map from user space. I have not been able to find them catalogued under /sys/fs .

[tohojo]

Chris Ward @.***> writes:

xdp-loader status indicated that old eBPF programs were being left, so I ran ip link set dev enp25s0 xdpgeneric off which cleaned them up. I also added metadata

struct {
        __uint(priority, 1);
        __uint(XDP_PASS, 1);
} XDP_RUN_CONFIG(xdp_sock_prog);

This means call on to the next program if you return XDP_PASS.

xdp-loader status now shows what I expect, but I am still getting packets sent to user-space even though my eBPF program is like

What is it showing? Your program running first, followed by the on installed by the XSK code? In that case, your program will return XDP_PASS, execution will continue to the next one, and that will do the redirect into userspace...

If you don't want the XSK code to install its own program on the interface, you should pass the XSK_LIBXDP_FLAGS__INHIBIT_PROG_LOAD flag as part of the socket config...

[tjcw]

The behaviour is as if my program is running and returning XDP_PASS, but then some other program is running and doing something I don't expect with the packets. But I have no other program; if this is happening, where is it coming from and what code is it running ?

I set

    xsk_cfg.libxdp_flags = XSK_LIBXDP_FLAGS__INHIBIT_PROG_LOAD;

and got an error loading the program. The message is

libxdp: Attached prog 'xsk_def_prog' with priority 20 in dispatcher entry 'prog1' with fd 28
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 29
libxdp: Pinning multiprog fd 20 beneath /sys/fs/bpf/xdp/dispatch-2-456
libxdp: Pinned link for prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-456/prog0-link
libxdp: Pinned prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-456/prog0-prog
libxdp: Pinned link for prog xsk_def_prog at /sys/fs/bpf/xdp/dispatch-2-456/prog1-link
libxdp: Pinned prog xsk_def_prog at /sys/fs/bpf/xdp/dispatch-2-456/prog1-prog
libxdp: Released lock fd 29
libxdp: Replacing XDP fd 7 with 20 on ifindex 2
libbpf: Kernel error message: Active program does not match expected
libxdp: Error attaching XDP program to ifindex 2: File exists
libxdp: XDP already loaded on device
libxdp: Failed to attach dispatcher on ifindex 2: File exists
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 29
libxdp: Unpinning multiprog fd 20 beneath /sys/fs/bpf/xdp/dispatch-2-456
libxdp: Unpinned link for prog xdp_sock_prog from /sys/fs/bpf/xdp/dispatch-2-456/prog0-link
libxdp: Unpinned prog xdp_sock_prog from /sys/fs/bpf/xdp/dispatch-2-456/prog0-prog
libxdp: Unpinned link for prog xsk_def_prog from /sys/fs/bpf/xdp/dispatch-2-456/prog1-link
libxdp: Unpinned prog xsk_def_prog from /sys/fs/bpf/xdp/dispatch-2-456/prog1-prog
libxdp: Removed pin directory /sys/fs/bpf/xdp/dispatch-2-456
libxdp: Released lock fd 29
xsk_setup_xdp_prog returns -17
ERROR: Can't setup AF_XDP socket 0 "File exists"
root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter#

[tjcw]

I found the default program; it is

int xsk_def_prog(struct xdp_md *ctx)
{
        /* A set entry here means that the corresponding queue_id
         * has an active AF_XDP socket bound to it.
         */
        return bpf_redirect_map(&xsks_map, ctx->rx_queue_index, XDP_PASS);
}

(which does not do what I eventually want, so I will need to make it not load). But this does not explain the behavour I am seeing; I have a test program which returns XDP_PASS for everything, so this combinatiion should redirect everything to user space. What's actually happening is that ICMP packets are going to user space and TCP packets are being passed throough to the kernel network stack. I will recheck whether I have the ethernet device set to direct packets to the queue I expect them to go to.

[tjcw]

My problem is with the queue the adapter is putting packets in, and it looks like a command such as

ethtool -N enp25s0 flow-type tcp4 action 0

does not work for my adapter. It gives messages

root@r28b29-n10:/home/tjcw# ethtool -N enp25s0 flow-type tcp4 action 0
rxclass: Cannot get RX class rule count: Operation not supported
Cannot insert classification rule
root@r28b29-n10:/home/tjcw#

which I had thought were warnings only, but it looks like packets are not being steered to queue 0.

[tohojo]

Chris Ward @.***> writes:

The behaviour is as if my program is running and returning XDP_PASS, but then some other program is running and doing something I don't expect with the packets. But I have no other program; if this is happening, where is it coming from and what code is it running ?

The XSK code will load a program unless you tell it not to (via that flag). You should be able to see that with xdp-loader status

I set

  xsk_cfg.libxdp_flags = XSK_LIBXDP_FLAGS__INHIBIT_PROG_LOAD;

and got an error loading the program. The message is

libxdp: Attached prog 'xsk_def_prog' with priority 20 in dispatcher entry 'prog1' with fd 28
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 29
libxdp: Pinning multiprog fd 20 beneath /sys/fs/bpf/xdp/dispatch-2-456
libxdp: Pinned link for prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-456/prog0-link
libxdp: Pinned prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-456/prog0-prog
libxdp: Pinned link for prog xsk_def_prog at /sys/fs/bpf/xdp/dispatch-2-456/prog1-link
libxdp: Pinned prog xsk_def_prog at /sys/fs/bpf/xdp/dispatch-2-456/prog1-prog

xdp_sock_prog is yours (presumably), and xsk_def_prog is the default program that the XSK code is loading. So whatever you're doing with that flag it's not working...

libxdp: Released lock fd 29 libxdp: Replacing XDP fd 7 with 20 on ifindex 2

This is a bit odd; it indicates that libxdp's notion of which program is attached to the kernel interface gets out of sync with what's actually in the kernel. Are you doing some other manipulation of XDP programs while this operation is going on?

Hard to tell from the partial log and without looking at your full code example what's going on here...

[tjcw]

My code is here https://github.com/tjcw/xdp-tutorial/tree/master/ebpf-filter ; you may need to flip around

//  xsk_cfg.libxdp_flags = XSK_LIBXDP_FLAGS__INHIBIT_PROG_LOAD;
    xsk_cfg.libxdp_flags = 0;

to see how I am trying to apply the flag. The code is revised from the advanced03-AF_XDP test case; I think I have the changes for BPF 1.0 and from BPF to XDP right, but this is the area I will check.

[tjcw]

Does XSK_LIBXDP_FLAGS__INHIBIT_PROG_LOAD go in xdp_flags or libxdp_flags ? I may have that wrong. Looking at the source for libxdp, I think I have it right.

[tjcw]

Here is the complete trace. I am not doing other manipulations of XDP programs during this, and I ran ip link set dev enp25s0 xdpgeneric off just before to make sure there were no XDP programs loaded on the interface. This is on Ubuntu 22.04 with 5.15.0-48-generic kernel.

root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter# ip link set dev 
enp25s0 xdpgeneric off
root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter# ./af_xdp_user -S -d enp25s0 -Q 0 --filename ./af_xdp_kern_filter.o
Opening program file ./af_xdp_kern_filter.o
libbpf: elf: skipping unrecognized data section(8) .xdp_run_config
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found .xdp_run_config
xdp_prog=0x55ff7f61e890
libxdp: Generating multi-prog dispatcher for 1 programs
libxdp: Looking for './xdp-dispatcher.o'
libxdp: Looking for '/usr/local/lib/bpf/xdp-dispatcher.o'
libxdp: Loading XDP program from '/usr/local/lib/bpf/xdp-dispatcher.o' section 'xdp_dispatcher'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: xdp_metadata_section=xdp_metadata
libxdp: btf_get_datasec looking for xdp_metadata found .rodata
libxdp: btf_get_datasec looking for xdp_metadata found license
libxdp: btf_get_datasec looking for xdp_metadata found xdp_metadata
libxdp: Verified XDP dispatcher version 1 <= 1
libxdp: Loading multiprog dispatcher for 1 programs
libxdp: Loaded XDP program xdp_dispatcher, got fd 5
libxdp: Checking dispatcher compatibility
libxdp: Looking for './xdp-dispatcher.o'
libxdp: Looking for '/usr/local/lib/bpf/xdp-dispatcher.o'
libxdp: Loading XDP program from '/usr/local/lib/bpf/xdp-dispatcher.o' section 'xdp_pass'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: Loaded XDP program xdp_pass, got fd 11
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 14
libxdp: Released lock fd 14
libxdp: Linking prog xdp_sock_prog as multiprog entry 0
libxdp: Loaded XDP program xdp_sock_prog, got fd 12
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found .xdp_run_config
libxdp: Attached prog 'xdp_sock_prog' with priority 1 in dispatcher entry 'prog0' with fd 15
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 16
libxdp: Pinning multiprog fd 7 beneath /sys/fs/bpf/xdp/dispatch-2-504
libxdp: Pinned link for prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-504/prog0-link
libxdp: Pinned prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-504/prog0-prog
libxdp: Released lock fd 16
libxdp: Replacing XDP fd -1 with 7 on ifindex 2
libxdp: Loaded 1 programs on ifindex '2' in skb mode
xsk_socket__create returns -16
ERROR: Can't setup AF_XDP socket 0 "Device or resource busy"
root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter# 

The XDP program is loded with

        xdp_prog=xdp_program__open_file(cfg.filename,NULL, NULL)  ;
        fprintf(stderr,"xdp_prog=%p\n", xdp_prog) ;
        err=xdp_program__attach(xdp_prog,
                cfg.ifindex, XDP_MODE_SKB, 0);

in the test case.

[tjcw]

I found one problem; I was running the application in another window as well. But it still doesn't work; here is the trace

root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter# ip link set dev enp25s0 xdpgeneric off
root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter# ./af_xdp_user -S -d enp25s0 -Q 0 --filename ./af_xdp_kern_filter.o
Opening program file ./af_xdp_kern_filter.o
libbpf: elf: skipping unrecognized data section(8) .xdp_run_config
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found .xdp_run_config
xdp_prog=0x55c4421c7890
libxdp: Generating multi-prog dispatcher for 1 programs
libxdp: Looking for './xdp-dispatcher.o'
libxdp: Looking for '/usr/local/lib/bpf/xdp-dispatcher.o'
libxdp: Loading XDP program from '/usr/local/lib/bpf/xdp-dispatcher.o' section 'xdp_dispatcher'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: xdp_metadata_section=xdp_metadata
libxdp: btf_get_datasec looking for xdp_metadata found .rodata
libxdp: btf_get_datasec looking for xdp_metadata found license
libxdp: btf_get_datasec looking for xdp_metadata found xdp_metadata
libxdp: Verified XDP dispatcher version 1 <= 1
libxdp: Loading multiprog dispatcher for 1 programs
libxdp: Loaded XDP program xdp_dispatcher, got fd 5
libxdp: Checking dispatcher compatibility
libxdp: Looking for './xdp-dispatcher.o'
libxdp: Looking for '/usr/local/lib/bpf/xdp-dispatcher.o'
libxdp: Loading XDP program from '/usr/local/lib/bpf/xdp-dispatcher.o' section 'xdp_pass'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: Loaded XDP program xdp_pass, got fd 11
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 14
libxdp: Released lock fd 14
libxdp: Linking prog xdp_sock_prog as multiprog entry 0
libxdp: Loaded XDP program xdp_sock_prog, got fd 12
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found .xdp_run_config
libxdp: Attached prog 'xdp_sock_prog' with priority 1 in dispatcher entry 'prog0' with fd 15
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 16
libxdp: Pinning multiprog fd 7 beneath /sys/fs/bpf/xdp/dispatch-2-583
libxdp: Pinned link for prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-583/prog0-link
libxdp: Pinned prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-583/prog0-prog
libxdp: Released lock fd 16
libxdp: Replacing XDP fd -1 with 7 on ifindex 2
libxdp: Loaded 1 programs on ifindex '2' in skb mode
xsk_socket__create returns 0
libxdp: xdp_metadata_section=xdp_metadata
libxdp: btf_get_datasec looking for xdp_metadata found .rodata
libxdp: btf_get_datasec looking for xdp_metadata found license
libxdp: btf_get_datasec looking for xdp_metadata found xdp_metadata
libxdp: Verified XDP dispatcher version 1 <= 1
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 7
libxdp: Reading multiprog component programs from pinned directory
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found .xdp_run_config
libxdp: Released lock fd 7
libxdp: Found multiprog with id 583 and 1 component progs
libxdp: Looking for './xsk_def_xdp_prog.o'
libxdp: Looking for '/usr/local/lib/bpf/xsk_def_xdp_prog.o'
libxdp: Loading XDP program from '/usr/local/lib/bpf/xsk_def_xdp_prog.o' section '(unknown)'
libbpf: elf: skipping unrecognized data section(7) .xdp_run_config
libbpf: elf: skipping unrecognized data section(8) xdp_metadata
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found .xdp_run_config
libxdp: xdp_metadata_section=xdp_metadata
libxdp: btf_get_datasec looking for xdp_metadata found .rodata
libxdp: btf_get_datasec looking for xdp_metadata found license
libxdp: btf_get_datasec looking for xdp_metadata found xdp_metadata
libxdp: Verified XDP dispatcher version 1 <= 1
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 15
libxdp: Reading multiprog component programs from pinned directory
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found .xdp_run_config
libxdp: Released lock fd 15
libxdp: Found multiprog with id 583 and 1 component progs
libxdp: Generating multi-prog dispatcher for 2 programs
libxdp: Looking for './xdp-dispatcher.o'
libxdp: Looking for '/usr/local/lib/bpf/xdp-dispatcher.o'
libxdp: Loading XDP program from '/usr/local/lib/bpf/xdp-dispatcher.o' section 'xdp_dispatcher'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: xdp_metadata_section=xdp_metadata
libxdp: btf_get_datasec looking for xdp_metadata found .rodata
libxdp: btf_get_datasec looking for xdp_metadata found license
libxdp: btf_get_datasec looking for xdp_metadata found xdp_metadata
libxdp: Verified XDP dispatcher version 1 <= 1
libxdp: Loading multiprog dispatcher for 2 programs
libxdp: Loaded XDP program xdp_dispatcher, got fd 18
libxdp: Checking dispatcher compatibility
libxdp: Looking for './xdp-dispatcher.o'
libxdp: Looking for '/usr/local/lib/bpf/xdp-dispatcher.o'
libxdp: Loading XDP program from '/usr/local/lib/bpf/xdp-dispatcher.o' section 'xdp_pass'
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found license
libxdp: btf_get_datasec looking for .xdp_run_config found xdp_metadata
libxdp: DATASEC '.xdp_run_config' not found.
libxdp: Loaded XDP program xdp_pass, got fd 24
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 27
libxdp: Released lock fd 27
libxdp: Linking prog xdp_sock_prog as multiprog entry 0
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found .rodata
libxdp: btf_get_datasec looking for .xdp_run_config found .xdp_run_config
libxdp: Attached prog 'xdp_sock_prog' with priority 1 in dispatcher entry 'prog0' with fd 22
libxdp: Linking prog xsk_def_prog as multiprog entry 1
libxdp: Loaded XDP program xsk_def_prog, got fd 25
libxdp: btf_get_datasec looking for .xdp_run_config found .maps
libxdp: btf_get_datasec looking for .xdp_run_config found .xdp_run_config
libxdp: Attached prog 'xsk_def_prog' with priority 20 in dispatcher entry 'prog1' with fd 28
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 29
libxdp: Pinning multiprog fd 20 beneath /sys/fs/bpf/xdp/dispatch-2-593
libxdp: Pinned link for prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-593/prog0-link
libxdp: Pinned prog xdp_sock_prog at /sys/fs/bpf/xdp/dispatch-2-593/prog0-prog
libxdp: Pinned link for prog xsk_def_prog at /sys/fs/bpf/xdp/dispatch-2-593/prog1-link
libxdp: Pinned prog xsk_def_prog at /sys/fs/bpf/xdp/dispatch-2-593/prog1-prog
libxdp: Released lock fd 29
libxdp: Replacing XDP fd 7 with 20 on ifindex 2
libbpf: Kernel error message: Active program does not match expected
libxdp: Error attaching XDP program to ifindex 2: File exists
libxdp: XDP already loaded on device
libxdp: Failed to attach dispatcher on ifindex 2: File exists
libxdp: Acquired lock from /sys/fs/bpf/xdp with fd 29
libxdp: Unpinning multiprog fd 20 beneath /sys/fs/bpf/xdp/dispatch-2-593
libxdp: Unpinned link for prog xdp_sock_prog from /sys/fs/bpf/xdp/dispatch-2-593/prog0-link
libxdp: Unpinned prog xdp_sock_prog from /sys/fs/bpf/xdp/dispatch-2-593/prog0-prog
libxdp: Unpinned link for prog xsk_def_prog from /sys/fs/bpf/xdp/dispatch-2-593/prog1-link
libxdp: Unpinned prog xsk_def_prog from /sys/fs/bpf/xdp/dispatch-2-593/prog1-prog
libxdp: Removed pin directory /sys/fs/bpf/xdp/dispatch-2-593
libxdp: Released lock fd 29
xsk_setup_xdp_prog returns -17
ERROR: Can't setup AF_XDP socket 0 "File exists"
root@r28b29-n10:/home/tjcw/workspace/xdp-tutorial/ebpf-filter#

The trace mentions

libxdp: Loading XDP program from '/usr/local/lib/bpf/xsk_def_xdp_prog.o' section '(unknown)'

but I thought it wasn't supposed to be loading this file.

[tjcw]

Looks like the problem is

    ret = xsk_setup_xdp_prog(
            slot == 0 ? cfg->ifindex : cfg->redirect_ifindex,
                    NULL) ;

which I had explicitly in my test case, and will load the default program. This is an error in my move from BPF to xsk; I will remove it.

[tjcw]

Yes, my test case now works as I want. I will close this issue; if anything else arises in moving to BPF 1.0 and xdp, I can open another one.