search

Xilinx-CNS / onload

OpenOnload high performance user-level network stack
Other
581 stars 95 forks source link

AF_XDP on ConnectX-5 #83

Closed boyracer2020 closed 1 year ago

boyracer2020 commented 2 years ago

Using onload (clone from repo on 06/20) I am seeing : 

oo:tcp_pingpong[2241]: netif_tcp_helper_alloc_u: ERROR: Failed to allocate stack (rc=-1)
See kernel messages in dmesg or /var/log/syslog for more details of this failure
oo:tcp_pingpong[2241]: netif_tcp_helper_alloc_u: ERROR: Failed to allocate stack (rc=-1)
See kernel messages in dmesg or /var/log/syslog for more details of this failure
Bound

and 

Jun 22 10:33:43 ubuntu-thinkcentre2 kernel: [ 3354.109987] [onload] [0]: WARNING: huge pages are incompatible with AF_XDP. Disabling hugepage support.
Jun 22 10:33:43 ubuntu-thinkcentre2 kernel: [ 3354.128545] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
Jun 22 10:33:43 ubuntu-thinkcentre2 kernel: [ 3354.128552] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP

looking thru the issues, I have enabled ntuples. No changes.

What's missing ? Do I need to install special MLNX drivers ?

OS: Ubuntu 20

thanks, M

shirshen12 commented 2 years ago

@boyracer2020 mention steps to reproduce the issue, what is your environment ? Bare-metal, Azure VMs or normal VMs running on KVM with PCI Passthrough with Mellanox ?

boyracer2020 commented 2 years ago

hi @shirshen12 , it's a bare-metal host running standard Ubuntu 20.04

shirshen12 commented 2 years ago

Ok, please see exact instructions to setup, please let me know if its works:

# upgrade to latest OS kernel
sudo apt update -y
sudo apt upgrade -y
sudo apt full-upgrade -y

# reboot into new kernel
reboot

sudo apt install build-essential net-tools unzip libcap-dev linux-tools-common linux-tools-generic netperf libevent-dev libnl-route-3-dev tk bison tcl libnl-3-dev flex libnl-route-3-200 dracut python2 libpcap-dev -y
sudo apt install initramfs-tools -y

# build the Mellanox driver, mlx5_core
wget --no-check-certificate https://content.mellanox.com/ofed/MLNX_OFED-5.6-1.0.3.3/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64.tgz
tar zxf MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64.tgz
cd MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64
sudo ./mlnxofedinstall --force --add-kernel-support --vma --tmpdir /mnt/
dracut -f

# reboot into new driver (optional)
reboot

# Install Onload:
git clone https://github.com/Xilinx-CNS/onload.git
cd onload
scripts/onload_mkdist --release
cd onload-<version>/scripts/
./onload_install
./onload_tool reload

# register the NiC with AF_XDP driver interface
echo enp2s0f0 > /sys/module/sfc_resource/afxdp/register
echo enp2s0f1 > /sys/module/sfc_resource/afxdp/register
boyracer2020 commented 2 years ago

did all of that , see below , still no luck :

Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.13.0-51-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

0 updates can be applied immediately.

Your Hardware Enablement Stack (HWE) is supported until April 2025.
Last login: Mon Jun 27 21:51:53 2022 from 192.168.0.102
fischer@munich-thinkcentre1:~$ sudo apt update -y
[sudo] password for fischer:
Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB]
Hit:2 http://de.archive.ubuntu.com/ubuntu focal InRelease
Get:3 http://de.archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
Get:4 http://security.ubuntu.com/ubuntu focal-security/main amd64 DEP-11 Metadata [40,7 kB]
Get:5 http://de.archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB]
Get:6 http://security.ubuntu.com/ubuntu focal-security/universe amd64 DEP-11 Metadata [66,3 kB]
Get:7 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 DEP-11 Metadata [2.464 B]
Get:8 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1.935 kB]
Get:9 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 DEP-11 Metadata [278 kB]
Get:10 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [15,6 kB]
Get:11 http://de.archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB]
Get:12 http://de.archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB]
Get:13 http://de.archive.ubuntu.com/ubuntu focal-updates/universe amd64 DEP-11 Metadata [391 kB]
Get:14 http://de.archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 DEP-11 Metadata [944 B]
Get:15 http://de.archive.ubuntu.com/ubuntu focal-backports/main amd64 DEP-11 Metadata [7.964 B]
Get:16 http://de.archive.ubuntu.com/ubuntu focal-backports/universe amd64 DEP-11 Metadata [30,5 kB]
Fetched 4.706 kB in 3s (1.619 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
fischer@munich-thinkcentre1:~$ sudo apt upgrade -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libfwupdplugin1 libpcre2-32-0
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  ubuntu-advantage-tools
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 876 kB of archives.
After this operation, 21,5 kB of additional disk space will be used.
Get:1 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 ubuntu-advantage-tools amd64 27.9~20.04.1 [876 kB]
Fetched 876 kB in 1s (1.079 kB/s)
Preconfiguring packages ...
(Reading database ... 176920 files and directories currently installed.)
Preparing to unpack .../ubuntu-advantage-tools_27.9~20.04.1_amd64.deb ...
Unpacking ubuntu-advantage-tools (27.9~20.04.1) over (27.8~20.04.1) ...
Setting up ubuntu-advantage-tools (27.9~20.04.1) ...
Installing new version of config file /etc/ubuntu-advantage/uaclient.conf ...
Created symlink /etc/systemd/system/multi-user.target.wants/ubuntu-advantage.service → /lib/systemd/system/ubuntu-advantage.service.
Processing triggers for man-db (2.9.1-1) ...
fischer@munich-thinkcentre1:~$ sudo apt full-upgrade -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libfwupdplugin1 libpcre2-32-0
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
fischer@munich-thinkcentre1:~$ sudo apt install build-essential net-tools unzip libcap-dev linux-tools-common linux-tools-generic netperf libevent-dev libnl-route-3-dev tk bison tcl libnl-3-dev flex libnl-route-3-200 dracut python2 libpcap-dev -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
bison is already the newest version (2:3.5.1+dfsg-1).
flex is already the newest version (2.6.4-6.2).
libcap-dev is already the newest version (1:2.32-1).
libevent-dev is already the newest version (2.1.11-stable-1).
libnl-3-dev is already the newest version (3.4.0-1).
libnl-route-3-200 is already the newest version (3.4.0-1).
libnl-route-3-dev is already the newest version (3.4.0-1).
libpcap-dev is already the newest version (1.9.1-3).
net-tools is already the newest version (1.60+git20180626.aebd88e-1ubuntu1).
unzip is already the newest version (6.0-25ubuntu1).
python2 is already the newest version (2.7.17-2ubuntu4).
tcl is already the newest version (8.6.9+1).
tk is already the newest version (8.6.9+1).
netperf is already the newest version (2.6.0-2.1).
build-essential is already the newest version (12.8ubuntu1.1).
linux-tools-common is already the newest version (5.4.0-121.137).
The following packages were automatically installed and are no longer required:
  libfwupdplugin1 libpcre2-32-0
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  linux-tools-5.4.0-121 linux-tools-5.4.0-121-generic
Suggested packages:
  dracut-network
The following packages will be REMOVED:
  initramfs-tools
The following NEW packages will be installed:
  dracut linux-tools-5.4.0-121 linux-tools-5.4.0-121-generic linux-tools-generic
0 upgraded, 4 newly installed, 1 to remove and 0 not upgraded.
Need to get 5.587 kB/5.590 kB of archives.
After this operation, 25,9 MB of additional disk space will be used.
Get:1 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-tools-5.4.0-121 amd64 5.4.0-121.137 [5.582 kB]
Get:2 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-tools-5.4.0-121-generic amd64 5.4.0-121.137 [1.996 B]
Get:3 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-tools-generic amd64 5.4.0.121.122 [2.428 B]
Fetched 5.587 kB in 4s (1.522 kB/s)
dpkg: initramfs-tools: dependency problems, but removing anyway as you requested:
 cryptsetup-initramfs depends on initramfs-tools (>= 0.129) | linux-initramfs-tool; however:
  Package initramfs-tools is to be removed.
  Package linux-initramfs-tool is not installed.
  Package initramfs-tools which provides linux-initramfs-tool is to be removed.
  Package dracut which provides linux-initramfs-tool is not installed.
 cryptsetup-initramfs depends on initramfs-tools (>= 0.129) | linux-initramfs-tool; however:
  Package initramfs-tools is to be removed.
  Package linux-initramfs-tool is not installed.
  Package initramfs-tools which provides linux-initramfs-tool is to be removed.
  Package dracut which provides linux-initramfs-tool is not installed.

(Reading database ... 176920 files and directories currently installed.)
Removing initramfs-tools (0.136ubuntu6.7) ...
Selecting previously unselected package dracut.
(Reading database ... 176908 files and directories currently installed.)
Preparing to unpack .../dracut_048+80-2_all.deb ...
Unpacking dracut (048+80-2) ...
Selecting previously unselected package linux-tools-5.4.0-121.
Preparing to unpack .../linux-tools-5.4.0-121_5.4.0-121.137_amd64.deb ...
Unpacking linux-tools-5.4.0-121 (5.4.0-121.137) ...
Selecting previously unselected package linux-tools-5.4.0-121-generic.
Preparing to unpack .../linux-tools-5.4.0-121-generic_5.4.0-121.137_amd64.deb ...
Unpacking linux-tools-5.4.0-121-generic (5.4.0-121.137) ...
Selecting previously unselected package linux-tools-generic.
Preparing to unpack .../linux-tools-generic_5.4.0.121.122_amd64.deb ...
Unpacking linux-tools-generic (5.4.0.121.122) ...
Setting up dracut (048+80-2) ...
dracut: Generating /boot/initrd.img-5.13.0-30-generic
dracut: Generating /boot/initrd.img-5.13.0-51-generic
Setting up linux-tools-5.4.0-121 (5.4.0-121.137) ...
Setting up linux-tools-5.4.0-121-generic (5.4.0-121.137) ...
Setting up linux-tools-generic (5.4.0.121.122) ...
Processing triggers for man-db (2.9.1-1) ...
fischer@munich-thinkcentre1:~$ sudo apt install initramfs-tools -y
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libfwupdplugin1 libpcre2-32-0
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  dracut
The following NEW packages will be installed:
  initramfs-tools
0 upgraded, 1 newly installed, 1 to remove and 0 not upgraded.
Need to get 9.248 B of archives.
After this operation, 122 kB of additional disk space will be used.
Get:1 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 initramfs-tools all 0.136ubuntu6.7 [9.248 B]
Fetched 9.248 B in 0s (26,4 kB/s)
dpkg: dracut: dependency problems, but removing anyway as you requested:
 cryptsetup-initramfs depends on initramfs-tools (>= 0.129) | linux-initramfs-tool; however:
  Package initramfs-tools is not installed.
  Package linux-initramfs-tool is not installed.
  Package initramfs-tools which provides linux-initramfs-tool is not installed.
  Package dracut which provides linux-initramfs-tool is to be removed.

(Reading database ... 176943 files and directories currently installed.)
Removing dracut (048+80-2) ...
Selecting previously unselected package initramfs-tools.
(Reading database ... 176939 files and directories currently installed.)
Preparing to unpack .../initramfs-tools_0.136ubuntu6.7_all.deb ...
Unpacking initramfs-tools (0.136ubuntu6.7) ...
Setting up initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: Generating /boot/initrd.img-5.13.0-51-generic
fischer@munich-thinkcentre1:~$ cd MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64/
fischer@munich-thinkcentre1:~/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64$ sudo ./mlnxofedinstall --force --add-kernel-support --vma --tmpdir /mnt/
Note: This program will create MLNX_OFED_LINUX TGZ for ubuntu20.04 under /mnt/MLNX_OFED_LINUX-5.6-1.0.3.3-5.13.0-51-generic directory.
See log file /mnt/MLNX_OFED_LINUX-5.6-1.0.3.3-5.13.0-51-generic/mlnx_iso.23702_logs/mlnx_ofed_iso.23702.log

Checking if all needed packages are installed...
Building MLNX_OFED_LINUX DEBS . Please wait...
Creating metadata-rpms for 5.13.0-51-generic ...
WARNING: If you are going to configure this package as a repository, then please note
WARNING: that it is not signed, therefore, you need to set 'trusted=yes' in the sources.list file.
WARNING: Example: deb [trusted=yes] file:/<path to MLNX_OFED DEBS folder> ./
Created /mnt/MLNX_OFED_LINUX-5.6-1.0.3.3-5.13.0-51-generic/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-ext.tgz
Removing old packages...
Uninstalling the previous version of MLNX_OFED_LINUX

Installing /mnt/MLNX_OFED_LINUX-5.6-1.0.3.3-5.13.0-51-generic/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-ext
/mnt/MLNX_OFED_LINUX-5.6-1.0.3.3-5.13.0-51-generic/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-ext/mlnxofedinstall --force --without-dkms --force --vma --tmpdir /mnt
Logs dir: /mnt/MLNX_OFED_LINUX.390304.logs
General log file: /mnt/MLNX_OFED_LINUX.390304.logs/general.log

Below is the list of MLNX_OFED_LINUX packages that you have chosen
(some may have been added by the installer due to package dependencies):

ofed-scripts
mlnx-tools
mlnx-ofed-kernel-utils
mlnx-ofed-kernel-modules
iser-modules
isert-modules
srp-modules
rdma-core
libibverbs1
ibverbs-utils
ibverbs-providers
libibverbs-dev
libibverbs1-dbg
libibumad3
libibumad-dev
ibacm
librdmacm1
rdmacm-utils
librdmacm-dev
mstflint
ibdump
libibmad5
libibmad-dev
libopensm
opensm
opensm-doc
libopensm-devel
libibnetdisc5
infiniband-diags
mft
kernel-mft-modules
srptools
rshim
mlnx-ethtool
mlnx-iproute2
perftest
ibutils2
dump-pr
ibsim
ibsim-doc
dpcp
libvma
libvma-utils
libvma-dev
sockperf

This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, RDMA or Distribution IB packages will be removed.
Those packages are removed due to conflicts with MLNX_OFED_LINUX, do not reinstall them.

Checking SW Requirements...
Removing old packages...
Installing new packages
Installing ofed-scripts-5.6...
Installing mlnx-tools-5.2.0...
Installing mlnx-ofed-kernel-utils-5.6...
Installing mlnx-ofed-kernel-modules-5.6...
Installing iser-modules-5.6...
Installing isert-modules-5.6...
Installing srp-modules-5.6...
Installing rdma-core-56mlnx40...
Installing libibverbs1-56mlnx40...
Installing ibverbs-utils-56mlnx40...
Installing ibverbs-providers-56mlnx40...
Installing libibverbs-dev-56mlnx40...
Installing libibverbs1-dbg-56mlnx40...
Installing libibumad3-56mlnx40...
Installing libibumad-dev-56mlnx40...
Installing ibacm-56mlnx40...
Installing librdmacm1-56mlnx40...
Installing rdmacm-utils-56mlnx40...
Installing librdmacm-dev-56mlnx40...
Installing mstflint-4.16.1...
Installing ibdump-6.0.0...
Installing libibmad5-56mlnx40...
Installing libibmad-dev-56mlnx40...
Installing libopensm-5.11.0.MLNX20220418.fd3d650...
Installing opensm-5.11.0.MLNX20220418.fd3d650...
Installing opensm-doc-5.11.0.MLNX20220418.fd3d650...
Installing libopensm-devel-5.11.0.MLNX20220418.fd3d650...
Installing libibnetdisc5-56mlnx40...
Installing infiniband-diags-56mlnx40...
Installing mft-4.20.0...
Installing kernel-mft-modules-4.20.0...
Installing srptools-56mlnx40...
Installing rshim-2.0.6...
Installing mlnx-ethtool-5.15...
Installing mlnx-iproute2-5.16.0...
Installing perftest-4.5...
Installing ibutils2-2.1.1...
Installing dump-pr-1.0...
Installing ibsim-0.10...
Installing ibsim-doc-0.10...
Installing dpcp-1.1.25...
Installing libvma-9.5.2...
Installing libvma-utils-9.5.2...
Installing libvma-dev-9.5.2...
Installing sockperf-3.8...
Selecting previously unselected package mlnx-fw-updater.
(Reading database ... 176933 files and directories currently installed.)
Preparing to unpack .../mlnx-fw-updater_5.6-1.0.3.3_amd64.deb ...
Unpacking mlnx-fw-updater (5.6-1.0.3.3) ...
Setting up mlnx-fw-updater (5.6-1.0.3.3) ...

Added 'RUN_FW_UPDATER_ONBOOT=no to /etc/infiniband/openib.conf

Initializing...
Attempting to perform Firmware update...
Querying Mellanox devices firmware ...

Device #1:
----------

  Device Type:      ConnectX5
  Part Number:      MCX512A-ACA_Ax_Bx
  Description:      ConnectX-5 EN network interface card; 10/25GbE dual-port SFP28; PCIe3.0 x8; tall bracket; ROHS R6
  PSID:             MT_0000000080
  PCI Device Name:  06:00.0
  Base GUID:        98039b030094f5bc
  Base MAC:         98039b94f5bc
  Versions:         Current        Available
     FW             16.33.1048     16.33.1048
     PXE            3.6.0502       3.6.0502
     UEFI           14.26.0017     14.26.0017

  Status:           Up to date

Log File: /tmp/wWF5BTNCPS
Real log file: /mnt/MLNX_OFED_LINUX.390304.logs/fw_update.log
Device (06:00.0):
        06:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
        Link Width: x4 ( WARNING - device supports x8 )
        PCI Link Speed: 8GT/s

Device (06:00.1):
        06:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
        Link Width: x4 ( WARNING - device supports x8 )
        PCI Link Speed: 8GT/s

Installation passed successfully
To load the new driver, run:
/etc/init.d/openibd restart
fischer@munich-thinkcentre1:~/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64$ sudo /etc/init.d/openibd restart
[sudo] password for fischer:
Unloading HCA driver:                                      [  OK  ]
Loading HCA driver and Access Layer:                       [  OK  ]
fischer@munich-thinkcentre1:~/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64$ /sbin/ifconfig -a
enp0s31f6: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.103  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::fe94:a4b2:7896:7718  prefixlen 64  scopeid 0x20<link>
        ether 4c:cc:6a:c6:2e:87  txqueuelen 1000  (Ethernet)
        RX packets 12155  bytes 12421927 (12.4 MB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 6503  bytes 632972 (632.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  memory 0xf7500000-f7520000

enp1s0f0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 00:0f:53:62:6f:d0  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 16

enp1s0f1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 00:0f:53:62:6f:d1  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17

enp6s0f0np0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::93a7:baff:a09:30b5  prefixlen 64  scopeid 0x20<link>
        ether 98:03:9b:94:f5:bc  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 17  bytes 2634 (2.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp6s0f1np1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 98:03:9b:94:f5:bd  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1315  bytes 128304 (128.3 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1315  bytes 128304 (128.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wlp5s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.107  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::50eb:18d5:38a5:e978  prefixlen 64  scopeid 0x20<link>
        ether 74:e5:f9:55:3c:19  txqueuelen 1000  (Ethernet)
        RX packets 1399  bytes 148130 (148.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 218  bytes 26313 (26.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

fischer@munich-thinkcentre1:~/MLNX_OFED_LINUX-5.6-1.0.3.3-ubuntu20.04-x86_64$ cd
fischer@munich-thinkcentre1:~$ cd onload-xdp.git/
fischer@munich-thinkcentre1:~/onload-xdp.git$ cd scripts/
fischer@munich-thinkcentre1:~/onload-xdp.git/scripts$ ./onload_mkdist --release
onload_mkdist: ERROR: Nothing in the ChangeLog for this version
fischer@munich-thinkcentre1:~/onload-xdp.git/scripts$ cd ..
fischer@munich-thinkcentre1:~/onload-xdp.git$ dmesg | tail
[ 3061.701229] mlx5_core 0000:06:00.1: mlx5_pcie_event:304:(pid 400193): PCIe slot advertised sufficient power (25W).
[ 3061.713395] mlx5_core 0000:06:00.0: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[ 3061.905598] mlx5_core 0000:06:00.0: Supported tc offload range - chains: 4294967294, prios: 4294967295
[ 3061.925283] mlx5_core 0000:06:00.1: MLX5E: StrdRq(1) RqSz(8) StrdSz(2048) RxCqeCmprss(0)
[ 3062.129277] mlx5_core 0000:06:00.1: Supported tc offload range - chains: 4294967294, prios: 4294967295
[ 3062.234892] mlx5_core 0000:06:00.1 enp6s0f1np1: renamed from eth1
[ 3062.262390] mlx5_core 0000:06:00.0 enp6s0f0np0: renamed from eth0
[ 3062.688095] mlx5_core 0000:06:00.0 enp6s0f0np0: Link up
[ 3062.699928] IPv6: ADDRCONF(NETDEV_CHANGE): enp6s0f0np0: link becomes ready
[ 3063.005144] mlx5_core 0000:06:00.1 enp6s0f1np1: Link down
fischer@munich-thinkcentre1:~/onload-xdp.git$
fischer@munich-thinkcentre1:~/onload-xdp.git$ sudo /sbin/ifconfig enp6s0f0np0 10.0.0.1 netmask 255.255.255.0 up
fischer@munich-thinkcentre1:~/onload-xdp.git$ cd
fischer@munich-thinkcentre1:~$  ./tcp_pingpong  -s -l 10.0.0.1
bind: Cannot assign requested address
fischer@munich-thinkcentre1:~$ sudo /sbin/ifconfig enp6s0f0np0 10.0.0.1 netmask 255.255.255.0 up
fischer@munich-thinkcentre1:~$ ./tcp_pingpong  -s -l 10.0.0.1
^C
fischer@munich-thinkcentre1:~$ onload ./tcp_pingpong  -s -l 10.0.0.1
oo:tcp_pingpong[401091]: Using Onload   [0]
oo:tcp_pingpong[401091]: Copyright 2019-2022 Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks
^C
fischer@munich-thinkcentre1:~$ sudo /sbin/ifconfig enp6s0f0np0 10.0.0.1 netmask 255.255.255.0 up
fischer@munich-thinkcentre1:~$ onload ./tcp_pingpong  -s -l 10.0.0.1
oo:tcp_pingpong[401113]: Using Onload   [1]
oo:tcp_pingpong[401113]: Copyright 2019-2022 Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks
^C
fischer@munich-thinkcentre1:~$
fischer@munich-thinkcentre1:~$ echo enp6s0f0np0 > /sys/module/sfc_resource/afxdp/register
-bash: /sys/module/sfc_resource/afxdp/register: Permission denied
fischer@munich-thinkcentre1:~$ sudo su
root@munich-thinkcentre1:/home/fischer# echo enp6s0f0np0 > /sys/module/sfc_resource/afxdp/register
root@munich-thinkcentre1:/home/fischer# echo enp6s0f1np1 > /sys/module/sfc_resource/afxdp/register
root@munich-thinkcentre1:/home/fischer# exit
exit
fischer@munich-thinkcentre1:~$ sudo /sbin/ifconfig enp6s0f0np0 10.0.0.1 netmask 255.255.255.0 up
fischer@munich-thinkcentre1:~$ ./tcp_pingpong  -s -l 10.0.0.1
^C
fischer@munich-thinkcentre1:~$ onload ./tcp_pingpong  -s -l 10.0.0.1
oo:tcp_pingpong[401175]: netif_tcp_helper_alloc_u: ERROR: Failed to allocate stack (rc=-1)
See kernel messages in dmesg or /var/log/syslog for more details of this failure
oo:tcp_pingpong[401175]: netif_tcp_helper_alloc_u: ERROR: Failed to allocate stack (rc=-1)
See kernel messages in dmesg or /var/log/syslog for more details of this failure
^C
fischer@munich-thinkcentre1:~$ dmesg | tail
[ 3340.826170] [onload] [2]: WARNING: huge pages are incompatible with AF_XDP. Disabling hugepage support.
[ 3340.872019] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 3340.872025] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP
[ 3340.872029] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 3340.872030] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP
[ 3340.880341] [onload] [3]: WARNING: huge pages are incompatible with AF_XDP. Disabling hugepage support.
[ 3340.893582] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 3340.893587] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP
[ 3340.893591] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 3340.893592] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP
shirshen12 commented 2 years ago

Check if the driver version is indeed, 5.6.x for Mellanox

ethtool -i <ifname> or simply ethtool <ifname>

Paste o/p here

boyracer2020 commented 2 years ago

yep. it shows version: 5.6-1.0.3

Note that this also happens with SFC and the error hint refers to sfc efhw

can someone try to reproduce? the same happens on a 2nd system. Ubuntu 20.04 , mlx or sfc does not matter.

boyracer2020 commented 2 years ago

full output:

 ethtool -i enp6s0f0np0
driver: mlx5_core
version: 5.6-1.0.3
firmware-version: 16.33.1048 (MT_0000000080)
expansion-rom-version:
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes
ol-alexandra commented 2 years ago

fischer@munich-thinkcentre1:~$ onload ./tcp_pingpong -s -l 10.0.0.1 oo:tcp_pingpong[401175]: netif_tcp_helper_alloc_u: ERROR: Failed to allocate stack (rc=-1)

You should be root to use Onload on AF_XDP. -1 == -EPERM.

boyracer2020 commented 2 years ago

magic :) that was the missing part. thanks a bunch. Not seeing this in the docs.

shirshen12 commented 2 years ago

yay!!! @ol-alexandra is back!

shirshen12 commented 2 years ago

Yes you have to be root as of now.

shirshen12 commented 2 years ago

BTW @boyracer2020 any non-trivial mult-threaded app will have trouble running in full performance since Onload-on-AF_XDP lacks transparent binding to multiple queues.

boyracer2020 commented 2 years ago

thanks. since the subject is about AF_XDP on ConnectX-5, do we know how to further reduce latency? some IRQ/Napi settings?

this is what we are seeing in a latency benchmark (round trip / 2): standard tcp/ip:

fischer@ubuntu-thinkcentre2:~$  ./tcp_pingpong  -h 10.0.0.1 -l 10.0.0.2 -E 128 -i 1000000
Bound
Connected
Half Round-Trip latency benchmark [tcp]:
  Size 1:        9.20 us         (mean=11.06 median=11.04, 99%=12.09 max=219.84, std_dev=0.64)
  Size 2:        9.09 us         (mean=11.02 median=11.00, 99%=11.94 max=345.58, std_dev=0.84)
  Size 4:        8.98 us         (mean=11.04 median=11.01, 99%=11.99 max=120.43, std_dev=0.61)
  Size 8:        9.08 us         (mean=11.08 median=11.05, 99%=12.05 max=432.24, std_dev=0.82)
  Size 16:       9.05 us         (mean=11.03 median=11.01, 99%=12.09 max=150.62, std_dev=0.66)
  Size 32:       9.01 us         (mean=11.03 median=11.01, 99%=11.93 max=284.35, std_dev=0.72)
  Size 64:       9.25 us         (mean=11.12 median=11.08, 99%=12.14 max=651.26, std_dev=0.99)
  Size 128:      9.52 us         (mean=11.27 median=11.23, 99%=12.17 max=138.19, std_dev=0.63)

and now with AF_XPD enabled:

fischer@ubuntu-thinkcentre2:~$ sudo onload --profile=latency ./tcp_pingpong  -h 10.0.0.1 -l 10.0.0.2 -E 128 -i 1000000
oo:tcp_pingpong[2745]: Using Onload   [6]
oo:tcp_pingpong[2745]: Copyright 2019-2022 Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks
Bound
Connected
Half Round-Trip latency benchmark [tcp]:
  Size 1:        6.06 us         (mean=8.86 median=8.76, 99%=11.10 max=162.18, std_dev=1.20)
  Size 2:        5.94 us         (mean=8.85 median=8.75, 99%=11.11 max=122.13, std_dev=1.20)
  Size 4:        5.94 us         (mean=8.60 median=8.58, 99%=10.81 max=269.04, std_dev=1.14)
  Size 8:        6.02 us         (mean=8.52 median=8.45, 99%=10.74 max=160.91, std_dev=1.09)
  Size 16:       5.97 us         (mean=8.55 median=8.48, 99%=10.78 max=268.14, std_dev=1.12)
  Size 32:       5.92 us         (mean=8.59 median=8.54, 99%=10.80 max=134.33, std_dev=1.10)
  Size 64:       6.43 us         (mean=8.68 median=8.61, 99%=10.88 max=159.15, std_dev=1.11)
  Size 128:      6.30 us         (mean=8.80 median=8.72, 99%=11.21 max=127.79, std_dev=1.21)

Onload without AF_XDP shows a nice 1.1usecs here.

Q: is there a way to reduce RX / TX operations with AF_XDP ?

maciejj-xilinx commented 2 years ago

Hi @boyracer2020 ,

So how low latency you could get would be device specific. There are some standard ways to improve:

sudo ethtool -c tx-usecs 0 rx-usecs 0

Another item is adjusting interrupt affinity... I'd be vague here as we did most of the work with throughput in mind in regards to AF_XDP.

P.S. Thanks @shirshen12 and @ol-alexandra for your support.