search

amzn / amzn-drivers

Official AWS drivers repository for Elastic Network Adapter (ENA) and Elastic Fabric Adapter (EFA)
457 stars 176 forks source link

command parameter 46 is not supported #241

Closed wrfeewrtgaqwfrwaq closed 1 year ago

wrfeewrtgaqwfrwaq commented 2 years ago

So I'm trying to use openonload with ena using AF_XDP API. Unfortunately it doesn't seem to pass any traffic through and dmesg shows:

ena 0000:7e:00.0 eth0: Command parameter 46 is not supported
[sfc efhw] rxclass_get_dev_info: rxclass: Cannot get RX class rule count
[onload] oof_local_port_addr_fixup_wild: 1:2047 ERROR: FILTER TCP 172.31.0.132:45907 0.0.0.0:0 failed (-95)
davidarinzon commented 2 years ago

Hi @wrfeewrtgaqwfrwaq,

We are aware of the current lack of support of OpenOnload on ENA (See https://github.com/amzn/amzn-drivers/issues/206). At this point, a W/A has been suggested (https://github.com/Xilinx-CNS/onload/issues/28#issuecomment-1054526865), you may follow the ticket (https://github.com/Xilinx-CNS/onload/issues/28) for more information. We will try the W/A above and update.

Thanks!

amitbern-aws commented 2 years ago

Hi @wrfeewrtgaqwfrwaq,

We tried the W/A above and succeeded running OpenOnload with ENA using AF_XDP API

Instructions:

1. Kernel preparations (AmazonLinux2)

$ sudo yum install git make gcc flex bison elfutils-libelf-devel openssl-devel dwarves -y
$ git clone https://github.com/torvalds/linux.git ~/linux && cd ~/linux
$ git checkout -B 5.19 v5.19
$ yes "" | make oldconfig
$ ./scripts/config -e CONFIG_NET_VENDOR_AMAZON
$ ./scripts/config -m CONFIG_ENA_ETHERNET
$ ./scripts/config -e CONFIG_MTD
$ ./scripts/config --set-str CONFIG_LOCALVERSION "-onload_support"
$ yes "" | make -j$(nproc)
$ make modules -j$(nproc)
$ sudo make modules_install -j$(nproc)
$ sudo make install -j$(nproc)
$ echo "GRUB_DEFAULT=0" | sudo tee -a   /etc/default/grub
$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
$ sudo reboot
$ grep -E 'CONFIG_MTD=|CONFIG_ENA_ETHERNET=|CONFIG_NET_VENDOR_AMAZON='  /lib/modules/`uname -r`/build/.config
CONFIG_MTD=y
CONFIG_NET_VENDOR_AMAZON=y
CONFIG_ENA_ETHERNET=m

2. Install supported ENA driver (2.8.0)

3. Install OpenOnload

sudo yum install libcap-devel -y
$ git clone https://github.com/Xilinx-CNS/onload.git && cd onload
$ sudo scripts/onload_install (for debuging onload: sudo scripts/onload_install --debug)
$ sudo onload_tool reload

From this point forward, you can use onload command to run any performance util, for example:

$ sudo su
$ onload iperf -c 1.2.3.4 -M 64 -t 5000 -i 1

onload: WARNING: EF_TCP_SYNRECV_MAX=16384 and EF_MAX_ENDPOINTS=8192 are inconsistent.
onload: EF_TCP_SYNRECV_MAX is set to 16384 based on /proc/sys/net/ipv4/tcp_max_syn_backlog value and assuming up to 4 listening sockets in the Onload stack
onload: Too few endpoints requested: ~4 syn-receive states consume one endpoint.
oo:iperf[69395]: Using Onload 50f66a47 2022-10-06 master  [2]
oo:iperf[69395]: Copyright 2019-2022 Xilinx, 2006-2019 Solarflare Communications, 2002-2005 Level 5 Networks
WARNING: attempt to set TCP maxmimum segment size to 64 failed.
Setting the MSS may not be implemented on this OS.
------------------------------------------------------------
Client connecting to 1.2.3.4, TCP port 5001
TCP window size:  128 KByte (default)
------------------------------------------------------------
[  3] local 1.2.3.5 port 47083 connected with 1.2.3.4 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   329 MBytes  2.76 Gbits/sec
[  3]  1.0- 2.0 sec   322 MBytes  2.70 Gbits/sec
[  3]  2.0- 3.0 sec   323 MBytes  2.71 Gbits/sec
amitbern-aws commented 1 year ago

Hi @wrfeewrtgaqwfrwaq, Did you try running with instructions above?

m4ce commented 1 year ago

@amitbern-aws - can you elaborate on why CONFIG_MTD=y is required?

amitbern-aws commented 1 year ago

@m4ce in the past, we received some openonload warnings due to missing MTD symbols (although it is not mentioned in any release notes), you can try without it.