Problem for Running XDP on Virtio network device

[beirvin]

Hi Dear Fellows: When I do AF_XDP test on Virtio network device, I got a problem as below. So pls help to figure out this problem, I don't know so much regarding virtual network device.

Exception information: libbpf: can't get next link: Invalid argument libbpf: Kernel error message: virtio_net: Too few free TX rings available

Regarding linux sorce code can be found at virtio_net.c@ virtnet_xdp_set() function: static int virtnet_xdp_set(struct net_device dev, struct bpf_prog prog, struct netlink_ext_ack *extack) { ... ... / XDP requires extra queues for XDP_TX / if (curr_qp + xdp_qp > vi->max_queue_pairs) { NL_SET_ERR_MSG_MOD(extack, "Too few free TX rings available"); netdev_warn(dev, "request %i queues but max is %i\n", curr_qp + xdp_qp, vi->max_queue_pairs); return -ENOMEM; } ... ... }

[tohojo]

Smith Beirvin @.***> writes:

Hi Dear Fellows: When I do AF_XDP test on Virtio network device, I got a problem as below. So pls help to figure out this problem, I don't know so much regarding virtual network device.

Well, you already found the information in the driver: you need more virtual queues to be able to support XDP (or, alternatively, you can the number of queue pairs used by the kernel stack with ethtool -L).

The number of queues need to be raised from the hypervisor side. See, e.g., https://cloudblog.switch.ch/2016/09/06/tuning-virtualized-network-node-multi-queue-virtio-net/

[beirvin]

Smith Beirvin @.***> writes: Hi Dear Fellows: When I do AF_XDP test on Virtio network device, I got a problem as below. So pls help to figure out this problem, I don't know so much regarding virtual network device. Well, you already found the information in the driver: you need more virtual queues to be able to support XDP (or, alternatively, you can the number of queue pairs used by the kernel stack with ethtool -L). The number of queues need to be raised from the hypervisor side. See, e.g., https://cloudblog.switch.ch/2016/09/06/tuning-virtualized-network-node-multi-queue-virtio-net/

Thanks for reply. but it seems that command "ethtool -L" can't work on virtual network interface(result as below). vhost135:~$ sudo ethtool -L ens9 combined 256 Cannot set device channel parameters: Invalid argument

[tohojo]

Smith Beirvin @.***> writes:

Smith Beirvin @.***> writes: Hi Dear Fellows: When I do AF_XDP test on Virtio network device, I got a problem as below. So pls help to figure out this problem, I don't know so much regarding virtual network device. Well, you already found the information in the driver: you need more virtual queues to be able to support XDP (or, alternatively, you can the number of queue pairs used by the kernel stack with ethtool -L). The number of queues need to be raised from the hypervisor side. See, e.g., https://cloudblog.switch.ch/2016/09/06/tuning-virtualized-network-node-multi-queue-virtio-net/

Thanks for reply. but it seems that command "ethtool -L" can't work on virtual network interface(result as below). vhost135:~$ sudo ethtool -L ens9 combined 256 Cannot set device channel parameters: Invalid argument

It maybe be too big? You could try setting it to 1 just to be sure? But yeah, probably need config on the hypervisor side...

[beirvin]

sudo ethtool -L ens9 combined

Smith Beirvin @.***> writes:

Smith Beirvin @.***> writes: > Hi Dear Fellows: When I do AF_XDP test on Virtio network device, I got a problem as below. So pls help to figure out this problem, I don't know so much regarding virtual network device. > Well, you already found the information in the driver: you need more virtual queues to be able to support XDP (or, alternatively, you can the number of queue pairs used by the kernel stack with ethtool -L). The number of queues need to be raised from the hypervisor side. See, e.g., https://cloudblog.switch.ch/2016/09/06/tuning-virtualized-network-node-multi-queue-virtio-net/ Thanks for reply. but it seems that command "ethtool -L" can't work on virtual network interface(result as below). vhost135:~$ sudo ethtool -L ens9 combined 256 Cannot set device channel parameters: Invalid argument It maybe be too big? You could try setting it to 1 just to be sure? But yeah, probably need config on the hypervisor side...

I have tried, it dosen't work. I guess it due to the linux 5.4 kernel, not support XDP running on virtio network interface.

[beirvin]

@tohojo Here I also get anoter problem. When I do XDP running on wifi interface, then I do ifconfig down and ifconfig up the wifi interface, and the wifi network seems not work, and dosen't have the running flag, do you know why? wlan0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 inet 192.168.2.149 netmask 255.255.255.0 broadcast 192.168.2.255 inet6 fe80::fe:19ff:fe4b:ba19 prefixlen 64 scopeid 0x20 ether 48:5d:60:bb:06:f6 txqueuelen 1000 (Ethernet) RX packets 97513 bytes 33345701 (33.3 MB) RX errors 0 dropped 1314 overruns 0 frame 0 TX packets 6610 bytes 747394 (747.3 KB) TX errors 0 dropped 1428 overruns 0 carrier 0 collisions 0

[tohojo]

Smith Beirvin @.***> writes:

@tohojo Here I also get anoter problem. When I do XDP running on wifi interface, then I do ifconfig down and ifconfig up the wifi interface, and the wifi network seems not work, and dosen't have the running flag, do you know why?

No idea; is this related to XDP at all? I.e., does the same thing work if you haven't loaded any XDP program?

[beirvin]

Smith Beirvin @.***> writes: @tohojo Here I also get anoter problem. When I do XDP running on wifi interface, then I do ifconfig down and ifconfig up the wifi interface, and the wifi network seems not work, and dosen't have the running flag, do you know why? No idea; is this related to XDP at all? I.e., does the same thing work if you haven't loaded any XDP program?

I didn't try that. But the exception messege is from the linux kernel

[uablrek]

(assuming kvm/qemu) You must tell qemu to use more queues. Below is a script function I use to create parametes to kvm. You can't use it directly, but use it as a guideline;

# Mqueue setup. Needed for XDP with 0-copy
netY() {
        local nodeid=$1
        local n=$2
        local tap b0 b1 dev tmp

        dev=xcbr$n
        tap=${dev}_t$nodeid

        tmp=/tmp/tmp/xcnet
        mkdir -p $tmp
        cat > $tmp/$tap <<EOF
#! /bin/sh
echo $tap \$1 > $tmp/$tap.log
ip tuntap add \$1 mode tap user $USER
ip link set dev \$1 master $dev
ip link set up \$1
EOF
        chmod a+x $tmp/$tap

        b0=$(printf '%02x' $nodeid)
        b1=$n
        echo " -netdev tap,ifname=$tap,id=net$n,script=$tmp/$tap,queues=4,vhost=on"
        echo " -device virtio-net-pci,mq=on,vectors=6,netdev=net$n,mac=00:00:00:01:0$b1:$b0"
}

NOTE script= must be used (for some reason), so a script is generated.

Then in your VM you should see;

vm-001 ~ # ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       4
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       2
vm-001 ~ # ethtool -L eth1 combined 1
vm-001 ~ # ethtool -l eth1
Channel parameters for eth1:
Pre-set maximums:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       4
Current hardware settings:
RX:             n/a
TX:             n/a
Other:          n/a
Combined:       1

The ethtool -L eth1 combined 1 is to use just one AF_XDP socket.

(Earlier the "n/a" was "0". Don't know when that changed.)