search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.28k stars 21.47k forks source link

Question: Azure VM offering with accelerating networking using Mellanox nic type ConnectX-4 or ConnectX-5 #45303

Closed dcasota closed 4 years ago

dcasota commented 4 years ago

Using for example the Standard_DS3_v2 offering , includes the Accelerating Networking feature, but, it uses the nic type ConnectX-3® / Pro.

The vendor Mellanox describes its products in matrix rows with driver type (Linux,Windows), supported speeds, Ethernet: RoCE versions, VXLAN, GENEVE, SR-IOV Ethernet, iSER, N-VDS ENS.

The Azure docs https://docs.microsoft.com/en-us/azure/virtual-network/create-vm-accelerated-networking-powershell describes for a Windows VM the use of ConnectX-3 Ethernetadapter only.

How to found / is there already an Azure VM offering with accelerating networking specifically using a newer Mellanox nic type ConnectX®-4 / Lx or ConnectX®-5 / Ex?

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

Karishma-Tiwari-MSFT commented 4 years ago

Thanks for the question. We are currently investigating and will update you shortly.

SubhashVasarapu-MSFT commented 4 years ago

@dcasota , Appreciate your patience. There is no possibility for now for checking/selecting the Mellanox driver for specific VM size before deploying. And also from the reproduced scenario, I found that a basic VM SKU which supports accelerated networking is carrying the same Mellanox ConnectX-3 network adapter as the Standard E64s v3 (64 vcpus, 432 GiB memory) VM which is largest among the SKU. So, as of now Azure by default provide you with the Mellanox ConnectX-3 network adapter. If you would like to install the Mellanox driver manually then please go through this manual.

We are closing this issue for now.If there are further questions regarding this matter, please reply and we will gladly continue the discussion.

Disclaimer: This response contains a reference to a third-party World Wide Web site. Microsoft is providing this information as a convenience to you. Microsoft does not control these sites and has not tested any software or information found on these sites; therefore, Microsoft cannot make any representations regarding the quality, safety, or suitability of any software or information found there. There are inherent dangers in the use of any software found on the Internet, and Microsoft cautions you to make sure that you completely understand the risk before retrieving any software from the Internet

dcasota commented 4 years ago

Okay, thanks for clarifying. Really appreciate you answer! I switched from v2 to the Standard_E4s_v3 offering. From lessons learned, the Mellanox ConnectX-3 driver is included in newer product drivers. The Azure docs for a Windows Server Virtual Machine is really good. And in respect to the vendors' reference for other OS the Mellanox driver could be installed manually.

zhuguang11111 commented 4 years ago

Could Azure provides the Mellanox ConnectX-4 network adapter now?

bsmetzer commented 4 years ago

Fsv_2 series offers Mellanox ConnectX-4 beginning with the F8s_v2 size

harikalyank commented 4 years ago

I tried with Fsv_2 series still its support Mellanox X-3.Could you please help is there any series really supports X-4 or latest

nikhilno1 commented 4 years ago

In my experience, X-3 or X-4 gets assigned randomly which can change during VM restart. If you add multiple interfaces, you may get X-4.

harikalyank commented 4 years ago

after adding multiple interfaces I have got X-4M. Thank you !!

jaagotarakesh commented 3 years ago

Hi Is there a way to force using only connectX-4 only. Currently I see it get assigned randomly, as Nikhil also pointed out here. See few issues on azure platform with ubuntu-18.04 based VM images: Poor performance with testpmd on connectX-3. sudo ./testpmd -l 0,1,2,3 -w 3f8f:00:02.0 --vdev="net_vdev_netvsc0,iface=eth1" -- --port-topology=chained --nb-cores 3 --rxq=3 --txq=3 -i

testpmd> show port stats all

######################## NIC statistics for port 1 ######################## RX-packets: 3989290 RX-missed: 0 RX-bytes: 5585004670 RX-errors: 0 RX-nombuf: 0 TX-packets: 3989290 TX-errors: 0 TX-bytes: 5585004670

Throughput (since last show) Rx-pps: 88563 Rx-bps: 991910536 Tx-pps: 88563 Tx-bps: 991910536 ############################################################################ testpmd>

sunilvaltix commented 3 years ago

@dcasota , Appreciate your patience. There is no possibility for now for checking/selecting the Mellanox driver for specific VM size before deploying. And also from the reproduced scenario, I found that a basic VM SKU which supports accelerated networking is carrying the same Mellanox ConnectX-3 network adapter as the Standard E64s v3 (64 vcpus, 432 GiB memory) VM which is largest among the SKU. So, as of now Azure by default provide you with the Mellanox ConnectX-3 network adapter. If you would like to install the Mellanox driver manually then please go through this manual.

This manual is not useful for anything. This is what MS would need to do at the host level to provide support for different adapters. End users can't do anything with it.

Is there a hope for the Azure SDK API to provide the ability to specify the adapter? We have customers who run into intermittent crashes with older mlx4 driver used by connectX-3 adapter in Linux. mlx5 driver used by connectX-4 adapter has no such issues.

param1992 commented 3 years ago

We are also facing similar issues, where we would want to know on which instances/scenarios Connectx4/Connectx5 adapters would be attached to the VM. The Connectx3 adapter which uses the mlx4 driver is not really stable with DPDK. We would want to know that we could meet our scale requirements etc. It would be really helpful if we get list of instances that support Connectx4/Connectx5 adapters.

sunilvaltix commented 3 years ago

We are also facing similar issues, where we would want to know on which instances/scenarios Connectx4/Connectx5 adapters would be attached to the VM. The Connectx3 adapter which uses the mlx4 driver is not really stable with DPDK. We would want to know that we could meet our scale requirements etc. It would be really helpful if we get list of instances that support Connectx4/Connectx5 adapters.

What version of DPDK are you guys using?

param1992 commented 3 years ago

We are also facing similar issues, where we would want to know on which instances/scenarios Connectx4/Connectx5 adapters would be attached to the VM. The Connectx3 adapter which uses the mlx4 driver is not really stable with DPDK. We would want to know that we could meet our scale requirements etc. It would be really helpful if we get list of instances that support Connectx4/Connectx5 adapters.

What version of DPDK are you guys using?

We are using DPDK 20.05

dcasota commented 3 years ago

How to found / is there already an Azure VM offering with accelerating networking specifically using a newer Mellanox nic type ConnectX®-4 / Lx or ConnectX®-5 / Ex?

It works for a Standard_E4s_v3 virtual machine. The test system is a genV2 VMware Photon OS (Azure) flavor.

lspci -nn -v
1ac5:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] (rev 80)
        Subsystem: Mellanox Technologies Device [15b3:0190]
        Physical Slot: 2
        Flags: bus master, fast devsel, latency 0, NUMA node 0
        Memory at fe0100000 (64-bit, prefetchable) [size=1M]
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] MSI-X: Enable+ Count=8 Masked-
        Kernel driver in use: mlx5_core

f581:00:02.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016] (rev 80)
        Subsystem: Mellanox Technologies Device [15b3:0190]
        Physical Slot: 1
        Flags: bus master, fast devsel, latency 0, NUMA node 0
        Memory at fe0000000 (64-bit, prefetchable) [size=1M]
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [9c] MSI-X: Enable+ Count=8 Masked-
        Kernel driver in use: mlx5_core

uname -a
Linux photonos 5.10.61-1.ph4 #1-photon SMP Fri Sep 3 18:23:40 UTC 2021 x86_64 GNU/Linux