af_xdp panics

[renard]

I am trying to test this tool to benchmark UDP.

I run the following command on an isolated benchmark bare metal lab:

for m in af_packet net_conn udp_syscall raw_socket af_pcap af_xdp ; do  cmd="./go-pktgen --iface eth2 --dstip 198.18.0.152  --dstport 1515 --dstmac  c4:00:ad:af:4c:24  --srcip 198.18.0.101 --srcmac ac:1f:6b:a5:8e:12 --duration 10 --payloadsize 1472  --method $m --streams 8"; echo $cmd; $cmd; sleep 3; done
./go-pktgen --iface eth2 --dstip 198.18.0.152 --dstport 1515 --dstmac c4:00:ad:af:4c:24 --srcip 198.18.0.101 --srcmac ac:1f:6b:a5:8e:12 --duration 10 --payloadsize 1472 --method af_packet --streams 8
812849 packets/s (9845 Mb/s)
812911 packets/s (9845 Mb/s)
812887 packets/s (9845 Mb/s)
812873 packets/s (9845 Mb/s)
813048 packets/s (9847 Mb/s)
812847 packets/s (9845 Mb/s)
812795 packets/s (9844 Mb/s)
812845 packets/s (9845 Mb/s)
./go-pktgen --iface eth2 --dstip 198.18.0.152 --dstport 1515 --dstmac c4:00:ad:af:4c:24 --srcip 198.18.0.101 --srcmac ac:1f:6b:a5:8e:12 --duration 10 --payloadsize 1472 --method net_conn --streams 8
813181 packets/s (9849 Mb/s)
813230 packets/s (9849 Mb/s)
812861 packets/s (9845 Mb/s)
812817 packets/s (9844 Mb/s)
813219 packets/s (9849 Mb/s)
812798 packets/s (9844 Mb/s)
812901 packets/s (9845 Mb/s)
812797 packets/s (9844 Mb/s)
./go-pktgen --iface eth2 --dstip 198.18.0.152 --dstport 1515 --dstmac c4:00:ad:af:4c:24 --srcip 198.18.0.101 --srcmac ac:1f:6b:a5:8e:12 --duration 10 --payloadsize 1472 --method udp_syscall --streams 8
812952 packets/s (9846 Mb/s)
812912 packets/s (9845 Mb/s)
812844 packets/s (9845 Mb/s)
812835 packets/s (9845 Mb/s)
812840 packets/s (9845 Mb/s)
812864 packets/s (9845 Mb/s)
812845 packets/s (9845 Mb/s)
812777 packets/s (9844 Mb/s)
./go-pktgen --iface eth2 --dstip 198.18.0.152 --dstport 1515 --dstmac c4:00:ad:af:4c:24 --srcip 198.18.0.101 --srcmac ac:1f:6b:a5:8e:12 --duration 10 --payloadsize 1472 --method raw_socket --streams 8
813136 packets/s (9848 Mb/s)
813057 packets/s (9847 Mb/s)
812977 packets/s (9846 Mb/s)
812901 packets/s (9845 Mb/s)
812931 packets/s (9846 Mb/s)
812962 packets/s (9846 Mb/s)
812968 packets/s (9846 Mb/s)
813048 packets/s (9847 Mb/s)
./go-pktgen --iface eth2 --dstip 198.18.0.152 --dstport 1515 --dstmac c4:00:ad:af:4c:24 --srcip 198.18.0.101 --srcmac ac:1f:6b:a5:8e:12 --duration 10 --payloadsize 1472 --method af_pcap --streams 8
813311 packets/s (9850 Mb/s)
812861 packets/s (9845 Mb/s)
812880 packets/s (9845 Mb/s)
812775 packets/s (9844 Mb/s)
812828 packets/s (9844 Mb/s)
812855 packets/s (9845 Mb/s)
812938 packets/s (9846 Mb/s)
812795 packets/s (9844 Mb/s)
812761 packets/s (9844 Mb/s)
./go-pktgen --iface eth2 --dstip 198.18.0.152 --dstport 1515 --dstmac c4:00:ad:af:4c:24 --srcip 198.18.0.101 --srcmac ac:1f:6b:a5:8e:12 --duration 10 --payloadsize 1472 --method af_xdp --streams 8
panic: sendto failed with rc=18446744073709551615 and errno=6

goroutine 36 [running]:
github.com/asavie/xdp.(*Socket).Transmit(0xc0000cad00, {0xc0000aa800?, 0x4?, 0x0?})
    /root/go/pkg/mod/github.com/asavie/xdp@v0.3.4-0.20220212172814-56d71236a029/xdp.go:485 +0x1e5
github.com/atoonk/go-pktgen/pktgen.(*AFXdpSender).Send(0xc0001f2090, {0x7f1710, 0xc0000f5260})
    /root/src/go-pktgen/pktgen/af_xdp.go:88 +0x437
main.runTest.func2()
    /root/src/go-pktgen/main.go:225 +0x2d
created by main.runTest in goroutine 34
    /root/src/go-pktgen/main.go:224 +0x21e
panic: sendto failed with rc=18446744073709551615 and errno=6

goroutine 83 [running]:
github.com/asavie/xdp.(*Socket).Transmit(0xc0003c6000, {0xc0003c8400?, 0x4?, 0x0?})
    /root/go/pkg/mod/github.com/asavie/xdp@v0.3.4-0.20220212172814-56d71236a029/xdp.go:485 +0x1e5
github.com/atoonk/go-pktgen/pktgen.(*AFXdpSender).Send(0xc000396000, {0x7f1710, 0xc0003a6000})
    /root/src/go-pktgen/pktgen/af_xdp.go:88 +0x437
main.runTest.func2()
    /root/src/go-pktgen/main.go:225 +0x2d
created by main.runTest in goroutine 14
    /root/src/go-pktgen/main.go:224 +0x21e

eth2 has 8 tx queues and 8 rx queues:

ls /sys/class/net/eth2/queues/
rx-0  rx-1  rx-2  rx-3  rx-4  rx-5  rx-6  rx-7  tx-0  tx-1  tx-2  tx-3  tx-4  tx-5  tx-6  tx-7

ethtool -l eth2 
Channel parameters for eth2:
Pre-set maximums:
RX:     n/a
TX:     n/a
Other:      1
Combined:   63
Current hardware settings:
RX:     n/a
TX:     n/a
Other:      1
Combined:   8

ethtool -L eth2  tx 16
netlink error: requested channel count exceeds maximum (offset 36)
netlink error: Invalid argument

I am using Debian 11.

eth2 is:

02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

with ixgbe driver.

did I missed something?

TIA.

[atoonk]

Yah you're hitting the same issue as I've described at the bottom of this blog article

I Opened an issue here, hoping the author of the library can help https://github.com/asavie/xdp/issues/31

It works ok with most non Hardware nics. For now if you want to experiment with af_xdp you can try for example lo or veth, also see this script I used for test env

[renard]

Thanks for the quick feedback. However I am not sure I can send more packets since I am saturating the wire ;-)

[atoonk]

yah let's keep an eye on https://github.com/asavie/xdp/issues/31
would be awesome to have it work on all NICs.