Transmit error when using Intel nic / ixgbe driver

[atoonk]

This library works great on, for example, virtio_net drivers or even loopback. But when trying it on a bare-metal server with an ixgbe nic, I get this transmit error

root@guest:~/xdp/examples/sendudp# go run sendudp.go  -interface enp1s0f0
sending UDP packets from 192.168.111.10 (b2:96:81:75:b2:11) to 192.168.111.1 (ff:ff:ff:ff:ff:ff)...
panic: sendto failed with rc=18446744073709551615 and errno=22

goroutine 1 [running]:
github.com/asavie/xdp.(*Socket).Transmit(0xc000162000, {0xc00010a800?, 0x5fa3a9?, 0x2f?})
    /root/xdp/xdp.go:485 +0x205
main.main()
    /root/xdp/examples/sendudp/sendudp.go:134 +0xb47
exit status 2

the same example on the loopback interface on the same server works fine

root@guest:~/xdp/examples/sendudp# go run sendudp.go  -interface lo
sending UDP packets from 192.168.111.10 (b2:96:81:75:b2:11) to 192.168.111.1 (ff:ff:ff:ff:ff:ff)...
859712 packets/s (9917 Mb/s)
862208 packets/s (9946 Mb/s)

Any idea how to resolve this or what the root cause is? Below my nic info.

root@guest:~/xdp/examples/sendudp# ethtool -i enp1s0f0
driver: ixgbe
version: 6.5.0-15-generic
firmware-version: 0x800006da, 1.2527.0
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

root@guest:~/xdp/examples/sendudp# ethtool -l enp1s0f0
Channel parameters for enp1s0f0:
Pre-set maximums:
RX:     n/a
TX:     n/a
Other:      1
Combined:   12
Current hardware settings:
RX:     n/a
TX:     n/a
Other:      1
Combined:   12

root@guest:~/xdp/examples/sendudp# lspci  | grep Ethernet
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

[atoonk]

The above was on a vultr baremetal machine. I just tried again on a few bare metal servers from www.latitude.sh including one with an i40e driver. But same challenges. Using lo worked fine on the same machine

root@c3-medium-x86-nyc-1:/home/ubuntu/xdp/examples/sendudp# ethtool -i eno1
driver: i40e
version: 5.15.0-94-lowlatency
firmware-version: 9.00 0x8000d28f 1.3295.0
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

root@c3-medium-x86-nyc-1:/home/ubuntu/xdp/examples/sendudp#  lspci | grep -i ether
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GBASE-T (rev 02)
05:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GBASE-T (rev 02)

root@c3-medium-x86-nyc-1:/home/ubuntu/xdp/examples/sendudp# go run sendudp.go  -interface eno1
sending UDP packets from 192.168.111.10 (b2:96:81:75:b2:11) to 192.168.111.1 (ff:ff:ff:ff:ff:ff)...
panic: sendto failed with rc=18446744073709551615 and errno=6

goroutine 1 [running]:
github.com/asavie/xdp.(*Socket).Transmit(0xc0000076c0, {0xc0000c6800?, 0x5f934e?, 0x2f?})
    /home/ubuntu/xdp/xdp.go:485 +0x205
main.main()
    /home/ubuntu/xdp/examples/sendudp/sendudp.go:134 +0xb47
exit status 2
root@c3-medium-x86-nyc-1:/home/ubuntu/xdp/examples/sendudp# go run sendudp.go  -interface lo
sending UDP packets from 192.168.111.10 (b2:96:81:75:b2:11) to 192.168.111.1 (ff:ff:ff:ff:ff:ff)...
1645696 packets/s (18984 Mb/s)
1635392 packets/s (18865 Mb/s)
^Csignal: interrupt

[slavc]

Hm, probably the drivers don't like what we're doing or how we're setting things up. Errno 22 is EINVAL ("Invalid argument") and errno 6 is ENXIO ("No such device or address"). I'll try to take a look at the source code of the drivers in the next couple of days.

[atoonk]

Awesome, thanks @slavc

[pstavirs]

Looking at the dmesg output may provide some more information about the errors.

[atoonk]

Thanks for the tip @pstavirs unfortunately nothing in dmesg or any of the /var/log files

root@c3-small-x86-mex2-1:/home/ubuntu/xdp/examples/sendudp# ethtool -i eno1
driver: ixgbe
version: 5.15.0-97-lowlatency
firmware-version: 0x80001743, 1.3082.0
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
root@c3-small-x86-mex2-1:/home/ubuntu/xdp/examples/sendudp# ip link show eno1
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 3c:ec:ef:28:9f:54 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f0

# Let's record the time, so we can compare in dmesg

root@c3-small-x86-mex2-1:/home/ubuntu/xdp/examples/sendudp# date ;go run sendudp.go  -interface eno1
Tue Feb 27 16:14:25 UTC 2024
sending UDP packets from 192.168.111.10 (b2:96:81:75:b2:11) to 192.168.111.1 (ff:ff:ff:ff:ff:ff)...
panic: sendto failed with rc=18446744073709551615 and errno=6

goroutine 1 [running]:
github.com/asavie/xdp.(*Socket).Transmit(0xc000112820, {0xc000142800?, 0x5df74b?, 0x2f?})
    /home/ubuntu/xdp/xdp.go:485 +0x1a5
main.main()
    /home/ubuntu/xdp/examples/sendudp/sendudp.go:134 +0xacb
exit status 2

# Checking dmesg, but note the no entries from around this time. the 2 items below were a few min earlier when system booted

root@c3-small-x86-mex2-1:/home/ubuntu/xdp/examples/sendudp# dmesg -T | tail -2
[Tue Feb 27 16:09:45 2024] systemd[1]: Started Journal Service.
[Tue Feb 27 16:09:46 2024] loop3: detected capacity change from 0 to 8
root@c3-small-x86-mex2-1:/home/ubuntu/xdp/examples/sendudp#

[atoonk]

Hi @slavc, I was curious if you had time to dig into this some more?
Also, I did a write up mentioning your awesome library

[atoonk]

@slavc just checking to see if you had time to look into this?