Onload on AF_XDP does not work on Mellanox NiCs in Azure

[shirshen12]

Hello,

I am trying to get the Onload to offload memcached on an Azure VM, but it does not work.

Below is the error message:

root@sriov-onload1 ~]# onload -p latency memcached -m 24576 -c 1024 -t 4 -u root -l <ip-address>:11211
oo:memcached[68979]: netif_tcp_helper_alloc_u: ENODEV.
This error can occur if:
 - no Solarflare network interfaces are active/UP, or they are running
   packed stream firmware or are disabled, and
 - there are no AF_XDP interfaces registered with sfc_resource

Environment details of Azure: VM Type: D8ds_v4 OS: Red Hat Enterprise Linux release 8.4 (Ootpa) Kernel version: 4.18.0-305.7.1.el8_4.x86_64 CPU Make: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz vCPUs: 8

VM has access to underlying NiC via SR-IOV, with mirror pass-through mode.

NiC Model: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] (rev 80) Kernel Driver : mlx5_core, version: 5.4-1.0.3 Kernel Driver Loaded ? YES SFC driver and Onload compiled and loaded: YES

sfc_char              110592  1 onload
sfc_resource          184320  2 onload,sfc_char
sfc                   655360  0
virtual_bus            16384  1 sfc
sfc_driverlink         16384  2 sfc,sfc_resource
mtd                    69632  1 sfc
mdio                   16384  1 sfc
vdpa                   16384  1 sfc

SR-IOV interface registered : YES, the below message comes only when we try to re-register the NiC interface.

[root@sriov-onload1 ~]# echo enP28349s1 > /sys/module/sfc_resource/afxdp/register
-bash: echo: write error: Operation already in progress.

dmesg o/p:

[    5.371350] mlx5_core 6ebd:00:02.0 enP28349s1: Link up
[    5.374995] hv_netvsc 000d3a7d-9c68-000d-3a7d-9c68000d3a7d eth0: Data path switched to VF: enP28349s1
[    5.377992] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    5.378043] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    6.454942] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: (null)
[    8.021253] fbcon: Taking over console
[    8.021576] Console: switching to colour frame buffer device 128x48
[   51.343609] hv_balloon: Max. dynamic memory size: 32768 MB
[  737.254449] Solarflare driverlink driver v5.3.9.1001 API v32.0
[  737.260284] Solarflare NET driver v5.3.9.1001
[  737.285720] Efx driverlink registering resource driver
[  737.334475] [onload] Onload <dev-snapshot>
[  737.334476] [onload] Copyright 2019-present Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks
[  737.439749] onload_cp_server[67516]: Spawned daemon process 67536
[  840.879660] [sfc efrm] efrm_nondl_register_device: register enP28349s1
[  840.879662] [sfc efrm] enP28349s1 type=4:
[  841.879298] [sfc efrm] enP28349s1 index=0 ifindex=3
[  841.879310] [onload] oo_nic_add: ifindex=3 oo_index=0
[  950.500153] [sfc efrm] efrm_nondl_register_device: register eth0
[  950.500156] [sfc efrm] eth0 type=4:
[  950.500796] hv_netvsc 000d3a7d-9c68-000d-3a7d-9c68000d3a7d eth0: XDP: not support LRO
[  950.500799] [sfc efrm] ?: ERROR: hardware init failed rc=-95
[  950.500800] [sfc efrm] eth0 index=1 ifindex=2
[  950.500810] [onload] oo_nic_add: ifindex=2 oo_index=1
[ 1601.579528] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 1601.579531] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP
[ 1749.414471] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 1749.414473] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP
[ 1789.819187] [sfc efhw] af_xdp_flush_rx_dma_channel: FIXME AF_XDP
[ 1789.819188] [sfc efhw] af_xdp_flush_tx_dma_channel: FIXME AF_XDP

Request your attention.

[maciejj-xilinx]

Hello,

Thanks for the detailed report.

Worth noting we have not experience on running Onload + AF_XDP with Mellanox NICs and Azure.

This is what I can devise:

[ 950.500799] [sfc efrm] ?: ERROR: hardware init failed rc=-95

Registration of the NIC apparently failed with EOPNOTSUPP (-95), that explains the problem seeing when starting the app.

The above dmesg message is preceded by this message from the driver:

[ 950.500796] hv_netvsc 000d3a7d-9c68-000d-3a7d-9c68000d3a7d eth0: XDP: not support LRO

My attempt to decode this message gives this:

| The network driver probably supports XDP, however, not when LRO is enabled |

Can LRO setting on the device be tweaked? It might be tricky in hyper-vised environment if this is per-port setting.

Let me know whether this helps,

Regards, Maciej

[shirshen12]

Hello @maciejj-xilinx ,

I disabled the LRO option on the Mellanox NiC and was able to successfully able to get Onload working.

To disable LRO on Mellanox NiCs in SR-IOV mode in Azure:

root@sriov-onload1 ~]# ethtool -k enP28349s1 | grep large-receive-offload
large-receive-offload: off [requested on]

root@sriov-onload1 ~]# ethtool -K enP28349s1 lro off

[root@sriov-onload1 ~]# ethtool -k enP28349s1 | grep large-receive-offload
large-receive-offload: off

Now, I am able to offload the application on Onload:

[root@sriov-onload1 ~]# onload -p latency memcached -m 24576 -c 1024 -t 4 -u root -l 10.113.65.35:11211
oo:memcached[613707]: Using Onload <dev-snapshot> [6]
oo:memcached[613707]: Copyright 2019-present Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks

Verified if onload_stackdump can see the stack created:

[root@sriov-onload1 ~]# onload_stackdump 
#stack-id stack-name      pids
6         -               -

Hence closing the issue and thanks for the support.