CentOS 7.7 HPC on Standard NC24rs_v3 - GPUs/IB devices missing

[tbugfinder]

Hi,

I'm using CentOS 7.7 HPC image as a source for a packer build. Within the first steps I'm running lspci. Unfortunately it doesn't include all 4 NVidia GPUs and the Mellanox IB device. VM type is: Standard NC24rs_v3 (24 vcpus, 448 GiB memory)

Anybody seen that before?

    azure-arm: + lspci
    azure-arm: 0000:00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled) (rev 03)
    azure-arm: 0000:00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 01)
    azure-arm: 0000:00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
    azure-arm: 0000:00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02)
    azure-arm: 0000:00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V virtual VGA
    azure-arm: 3130:00:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 PCIe 16GB] (rev a1)
    azure-arm: + source /etc/os-release

Using CentOS 7.7 HPC on HB60rs lists the Mellanox IB device:

    azure-arm: + lspci
    azure-arm: 0000:00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled) (rev 03)
    azure-arm: 0000:00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 01)
    azure-arm: 0000:00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
    azure-arm: 0000:00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02)
    azure-arm: 0000:00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V virtual VGA
    azure-arm: be1a:00:02.0 Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]

[tbugfinder]

I've deployed another set of VMs using CentOS-HPC 7.7 and CentOS 7.7 on Standard_NC24rs_v2. Those also list only a single GPU, but no IB device.

[szarkos]

Regarding the IB devices - the CentOS 7.6/7.7 HPC images are only designed to work on the newer HB and HC instance types where the IB interface is exposed to the VM via SRIOV. For NC24r I think you need to use an older CentOS image (7.5 or earlier) which include an older driver to work with other HPC instance types such as the A8/A9/NC24r/H-series/etc.

I would defer to @jithinjosepkl or @vermagit about whether there will be a GPU instance size that expose the IB interface via SRIOV to work with these new CentOS 7.6+ images - or you can file an issue at https://github.com/Azure/azhpc-images/ to reach the Azure HPC folks directly.

[szarkos]

This comment might be moot given my previous comment about the CentOS 7.6+ HPC images not supporting the IB interface on NC24r, however for completeness here is the reason why you can only see one GPU device on CentOS 7.7 -

Sadly I think this is a bug in the more recent RHEL/CentOS 7.7 kernels regarding how PCI devices are addressed. I'm not sure when this will be addressed upstream, but it can be fixed by installing newer Hyper-V drivers. The easiest way is to use the Azure-tuned kernel that is part of the CentOS Virtualization SIG (https://wiki.centos.org/SpecialInterestGroup/Virtualization) -

$ sudo yum install centos-release-azure $ sudo yum install kernel-azure kernel-azure-tools $ sudo reboot

Another way is to just install the Hyper-V drivers as a kmod:

$ sudo yum install microsoft-hyper-v $ sudo reboot

I think using kernel-azure is better and less error prone than the kmod drivers, particularly when it comes updating the kernel. In any case, if you only need to use NC24 (not NC24r) then hopefully this should help unblock things.

[tbugfinder]

My understanding is that NC24rs_v3 has SR-IOV enabled by now. https://azure.microsoft.com/en-us/updates/sr-iov-availability-schedule-on-ncv3-virtual-machines-sku/

[szarkos]

Ah, good news. I no longer work at MSFT so I guess I'm a little out of the loop on the new developments :) That also means that I don't have a test account to help repro, so I'll defer to the other folks mentioned on this issue for more investigation. You might also consider filing a support ticket as well to investigate all sides of the issue.

I still think installing the Azure-tuned kernel is a solution for the single-GPU issue, perhaps you can confirm?

The IB issue is a bit strange - I would expect even without the latest ofed or mlx drivers one should at least see the IB interface exposed as a PCI device. Does this issue repro consistently? It may also be useful to quickly test another distro such as Ubuntu just to see if it can just see the IB device via lspci.

[vermagit]

@tbugfinder What you have run into the PCI ID bug that @szarkos referred to above. While NC_v3 are SR-IOV enabled everywhere, using the CentOS-HPC and (vanilla) CentOS images will on the NC24rs_v3 will run into the above error. This will lead to lspci not showing all 4 GPUs and no IB. I haven't tested @szarkos's solution above.

However, Red Hat released the fix in kernel version 3.10.0-1062.12.1, which is available for RHEL 7.7 and CentOS 7.7. The CentOS 7.7 image in the Azure Marketplace has not been updated, but the new kernel version is available via the normal “yum update” mechanism.

Red Hat also released the fix in kernel version 3.10.0-957.43.1, which is for RHEL 7.6. However, CentOS 7.6 will not offer the updated kernel because the CentOS community stops providing updates to version x.n when x.n+1 becomes available.

[tbugfinder]

Thank you! After running yum update and booting into the new kernel lspci output is valid (4 GPUs and MLX adapter). 0000:00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (AGP disabled) (rev 03) 0000:00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 01) 0000:00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01) 0000:00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 02) 0000:00:08.0 VGA compatible controller: Microsoft Corporation Hyper-V virtual VGA 0001:00:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 PCIe 16GB] (rev a1) 0002:00:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 PCIe 16GB] (rev a1) 0003:00:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 PCIe 16GB] (rev a1) 0004:00:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 PCIe 16GB] (rev a1) 225e:00:02.0 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] `

Now I'm missing the ib0 network interface with a valid IP address.

[vermagit]

Was it with the -HPC or the vanilla CentOS? In any case, I'd imagine the OFED drivers would need to be installed. We'll incorporate this with an updated -HPC image soon. But in the meantime, could you try installing MOFED (with the extension) or better still manually?

[tbugfinder]

it doesn't matter, if CentOS 7.7 or CentOS-HPC 7.7 was used.

After running yum update and rebooting, ibstat, ibstatus report properly IB devices, so IMHO the OFED driver installations are valid.

Do you mean installing the extension "InfiniBandDriverLinux" -Publisher "Microsoft.HpcCompute"?

[vermagit]

I'll update here when an updated (and Gen 2) CentOS-HPC 7.7 is available in the marketplace. Also plan to fix/test the extension above works with the latest releases.

[tbugfinder]

I had to run yum update to fix GPU issues and install later version of MLNX drivers. This fixed mentioned issues.

[vermagit]

Yup, just need to go beyond 3.10.0-1062.12.1

[tbugfinder]

thank you!