search

AsBuiltReport / AsBuiltReport.VMware.vSphere

Repository for AsBuiltReport VMware vSphere module
https://www.asbuiltreport.com
MIT License
112 stars 39 forks source link

Unable to retrieve PCIe device information on some VMHosts #33

Closed tpcarman closed 4 years ago

tpcarman commented 5 years ago

Describe the bug Script terminates with the following error when trying to retrieve PCIe device information

New-AsBuiltReport : Message: EsxCLI.CLIFault.summary;
InnerText: There is no pnic with name vmnic2EsxCLI.CLIFault.summary
At line:1 char:1
+ New-AsBuiltReport -Report VMware.vSphere -Target vcenter.domain.local ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,New-AsBuiltReport

To Reproduce When connecting to vCenter with PowerCLI and running the Get-PciDeviceDetail function manually, the following information and errors are displayed

VMkernel Name    : vmhba0
Device Name      : Smart Array P220i
Driver           : nhpsa
Driver Version   : 2.0.24-1OEM.650.0.0.4598673
Firmware Version : N/A
VIB Name         : nhpsa
VIB Version      : 2.0.24-1OEM.650.0.0.4598673

VMkernel Name    : vmnic0
Device Name      : QLogic 57810 10 Gigabit Ethernet Adapter
Driver           : bnx2x
Driver Version   : Version 2.713.30.v60.9, Build: 2494585, Interface: 9.2 Built on: Sep 20 2017
Firmware Version : bc 7.13.75
VIB Name         : net-bnx2x
VIB Version      : 2.713.30.v60.9-1OEM.600.0.0.2494585

VMkernel Name    : vmnic1
Device Name      : QLogic 57810 10 Gigabit Ethernet Adapter
Driver           : bnx2x
Driver Version   : Version 2.713.30.v60.9, Build: 2494585, Interface: 9.2 Built on: Sep 20 2017
Firmware Version : bc 7.13.75
VIB Name         : net-bnx2x
VIB Version      : 2.713.30.v60.9-1OEM.600.0.0.2494585

Message: EsxCLI.CLIFault.summary;
InnerText: There is no pnic with name vmnic2EsxCLI.CLIFault.summary
At line:47 char:21
+ ...             $vmnicDetail = $esxcli.network.nic.get.Invoke(@{nicname = ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], MethodFault
    + FullyQualifiedErrorId : VMware.VimAutomation.Sdk.Types.V1.ErrorHandling.VimException.MethodFault

VMkernel Name    : vmnic2
Device Name      : QLogic 57810 10 Gigabit Ethernet Adapter
Driver           : bnx2x
Driver Version   : Version 2.713.30.v60.9, Build: 2494585, Interface: 9.2 Built on: Sep 20 2017
Firmware Version : bc 7.13.75
VIB Name         : net-bnx2x
VIB Version      : 2.713.30.v60.9-1OEM.600.0.0.2494585

Message: EsxCLI.CLIFault.summary;
InnerText: There is no pnic with name vmnic3EsxCLI.CLIFault.summary
At line:47 char:21
+ ...             $vmnicDetail = $esxcli.network.nic.get.Invoke(@{nicname = ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], MethodFault
    + FullyQualifiedErrorId : VMware.VimAutomation.Sdk.Types.V1.ErrorHandling.VimException.MethodFault

VMkernel Name    : vmnic3
Device Name      : QLogic 57810 10 Gigabit Ethernet Adapter
Driver           : bnx2x
Driver Version   : Version 2.713.30.v60.9, Build: 2494585, Interface: 9.2 Built on: Sep 20 2017
Firmware Version : bc 7.13.75
VIB Name         : net-bnx2x
VIB Version      : 2.713.30.v60.9-1OEM.600.0.0.2494585

VMkernel Name    : vmnic4
Device Name      : QLogic 57810 10 Gigabit Ethernet Adapter
Driver           : bnx2x
Driver Version   : Version 2.713.30.v60.9, Build: 2494585, Interface: 9.2 Built on: Sep 20 2017
Firmware Version : bc 7.13.75
VIB Name         : net-bnx2x
VIB Version      : 2.713.30.v60.9-1OEM.600.0.0.2494585

VMkernel Name    : vmnic5
Device Name      : QLogic 57810 10 Gigabit Ethernet Adapter
Driver           : bnx2x
Driver Version   : Version 2.713.30.v60.9, Build: 2494585, Interface: 9.2 Built on: Sep 20 2017
Firmware Version : bc 7.13.75
VIB Name         : net-bnx2x
VIB Version      : 2.713.30.v60.9-1OEM.600.0.0.2494585

Expected behavior PCIe device information should be retrieved successfully from all hosts which are in a connected state

tpcarman commented 5 years ago

@equelin @edmsanchez Any ideas as to what could be causing this issue? I haven't had the opportunity to investigate further as yet.

tpcarman commented 5 years ago 
PS C:\>$esxcli.network.nic.list.Invoke()

AdminStatus : Up
Description : Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter
Driver      : bnx2x
Duplex      : Full
Link        : Up
LinkStatus  : Up
MACAddress  : fc:15:b4:15:14:60
MTU         : 1500
Name        : vmnic0
PCIDevice   : 0000:04:00.0
Speed       : 10000

AdminStatus : Up
Description : Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter
Driver      : bnx2x
Duplex      : Full
Link        : Up
LinkStatus  : Up
MACAddress  : fc:15:b4:15:14:64
MTU         : 1500
Name        : vmnic1
PCIDevice   : 0000:04:00.1
Speed       : 10000

AdminStatus : Up
Description : Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter
Driver      : bnx2x
Duplex      : Full
Link        : Up
LinkStatus  : Up
MACAddress  : fc:15:b4:15:14:62
MTU         : 1500
Name        : vmnic4
PCIDevice   : 0000:04:00.4
Speed       : 10000

AdminStatus : Up
Description : Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter
Driver      : bnx2x
Duplex      : Full
Link        : Up
LinkStatus  : Up
MACAddress  : fc:15:b4:15:14:66
MTU         : 1500
Name        : vmnic5
PCIDevice   : 0000:04:00.5
Speed       : 10000

vmnic2 and vmnic3 are not listed as pNICs, however they have a VMkernel name associated as per below

C:\>$esxcli.hardware.pci.list.Invoke() | Where-Object {$_.VMKernelName -like "vmnic*"} | Sort-Object -Property VMKernelName

Address              : 0000:04:00.0
Bus                  : 4
Chassis              : 0
ConfiguredOwner      : Unknown
CurrentOwner         : VMkernel
DependentDevice      : PCI 0:4:0:0
DeviceClass          : 512
DeviceClassName      : Ethernet controller
DeviceID             : 5774
DeviceName           : QLogic 57810 10 Gigabit Ethernet Adapter
FPTSharable          : true
Flags                : 12801
Function             : 0
IRQ                  : 5
InterruptLine        : 5
InterruptVector      : 52
ModuleID             : 4129
ModuleName           : bnx2x
PCIPin               : 0
ParentDevice         : PCI 0:0:2:0
PassthruCapable      : true
PhysicalSlot         : 4294967295
ProgrammingInterface : 0
ResetMethod          : Function reset
RevisionID           : 16
Segment              : 0
Slot                 : 0
SlotDescription      : NIC Port 1
SpawnedBus           : 0
SubDeviceID          : 6450
SubVendorID          : 4156
VMkernelName         : vmnic0
VendorID             : 5348
VendorName           : Broadcom Corporation

Address              : 0000:04:00.1
Bus                  : 4
Chassis              : 0
ConfiguredOwner      : Unknown
CurrentOwner         : VMkernel
DependentDevice      : PCI 0:4:0:1
DeviceClass          : 512
DeviceClassName      : Ethernet controller
DeviceID             : 5774
DeviceName           : QLogic 57810 10 Gigabit Ethernet Adapter
FPTSharable          : true
Flags                : 12801
Function             : 1
IRQ                  : 7
InterruptLine        : 7
InterruptVector      : 53
ModuleID             : 4129
ModuleName           : bnx2x
PCIPin               : 1
ParentDevice         : PCI 0:0:2:0
PassthruCapable      : true
PhysicalSlot         : 4294967295
ProgrammingInterface : 0
ResetMethod          : Function reset
RevisionID           : 16
Segment              : 0
Slot                 : 0
SlotDescription      : NIC Port 2
SpawnedBus           : 0
SubDeviceID          : 6450
SubVendorID          : 4156
VMkernelName         : vmnic1
VendorID             : 5348
VendorName           : Broadcom Corporation

Address              : 0000:04:00.2
Bus                  : 4
Chassis              : 0
ConfiguredOwner      : Unknown
CurrentOwner         : VMkernel
DependentDevice      : PCI 0:4:0:2
DeviceClass          : 512
DeviceClassName      : Ethernet controller
DeviceID             : 5774
DeviceName           : QLogic 57810 10 Gigabit Ethernet Adapter
FPTSharable          : true
Flags                : 12801
Function             : 2
IRQ                  : 7
InterruptLine        : 7
InterruptVector      : 53
ModuleID             : 4129
ModuleName           : bnx2x
PCIPin               : 1
ParentDevice         : PCI 0:0:2:0
PassthruCapable      : true
PhysicalSlot         : 4294967295
ProgrammingInterface : 0
ResetMethod          : Function reset
RevisionID           : 16
Segment              : 0
Slot                 : 0
SlotDescription      : NIC Port 3
SpawnedBus           : 0
SubDeviceID          : 6450
SubVendorID          : 4156
VMkernelName         : vmnic2
VendorID             : 5348
VendorName           : Broadcom Corporation

Address              : 0000:04:00.3
Bus                  : 4
Chassis              : 0
ConfiguredOwner      : Unknown
CurrentOwner         : VMkernel
DependentDevice      : PCI 0:4:0:3
DeviceClass          : 512
DeviceClassName      : Ethernet controller
DeviceID             : 5774
DeviceName           : QLogic 57810 10 Gigabit Ethernet Adapter
FPTSharable          : true
Flags                : 12801
Function             : 3
IRQ                  : 10
InterruptLine        : 10
InterruptVector      : 54
ModuleID             : 4129
ModuleName           : bnx2x
PCIPin               : 2
ParentDevice         : PCI 0:0:2:0
PassthruCapable      : true
PhysicalSlot         : 4294967295
ProgrammingInterface : 0
ResetMethod          : Function reset
RevisionID           : 16
Segment              : 0
Slot                 : 0
SlotDescription      : NIC Port 4
SpawnedBus           : 0
SubDeviceID          : 6450
SubVendorID          : 4156
VMkernelName         : vmnic3
VendorID             : 5348
VendorName           : Broadcom Corporation

Address              : 0000:04:00.4
Bus                  : 4
Chassis              : 0
ConfiguredOwner      : Unknown
CurrentOwner         : VMkernel
DependentDevice      : PCI 0:4:0:4
DeviceClass          : 512
DeviceClassName      : Ethernet controller
DeviceID             : 5774
DeviceName           : QLogic 57810 10 Gigabit Ethernet Adapter
FPTSharable          : true
Flags                : 12801
Function             : 4
IRQ                  : 10
InterruptLine        : 10
InterruptVector      : 54
ModuleID             : 4129
ModuleName           : bnx2x
PCIPin               : 2
ParentDevice         : PCI 0:0:2:0
PassthruCapable      : true
PhysicalSlot         : 4294967295
ProgrammingInterface : 0
ResetMethod          : Function reset
RevisionID           : 16
Segment              : 0
Slot                 : 0
SlotDescription      : NIC Port 5
SpawnedBus           : 0
SubDeviceID          : 6450
SubVendorID          : 4156
VMkernelName         : vmnic4
VendorID             : 5348
VendorName           : Broadcom Corporation

Address              : 0000:04:00.5
Bus                  : 4
Chassis              : 0
ConfiguredOwner      : Unknown
CurrentOwner         : VMkernel
DependentDevice      : PCI 0:4:0:5
DeviceClass          : 512
DeviceClassName      : Ethernet controller
DeviceID             : 5774
DeviceName           : QLogic 57810 10 Gigabit Ethernet Adapter
FPTSharable          : true
Flags                : 12801
Function             : 5
IRQ                  : 10
InterruptLine        : 10
InterruptVector      : 55
ModuleID             : 4129
ModuleName           : bnx2x
PCIPin               : 3
ParentDevice         : PCI 0:0:2:0
PassthruCapable      : true
PhysicalSlot         : 4294967295
ProgrammingInterface : 0
ResetMethod          : Function reset
RevisionID           : 16
Segment              : 0
Slot                 : 0
SlotDescription      : NIC Port 6
SpawnedBus           : 0
SubDeviceID          : 6450
SubVendorID          : 4156
VMkernelName         : vmnic5
VendorID             : 5348
VendorName           : Broadcom Corporation
tpcarman commented 5 years ago

discovered vmnic2 and vmnic3 are FCoE adapters

C:\>$esxcli.fcoe.nic.list.Invoke()

Active            : true
FCOENICName       : vmnic2
PrioritySettable  : false
SourceMAC         : fc:15:b4:15:b3:d1
SourceMACSettable : false
UserPriority      : 3
VLANRangeSettable : true
VN2VNModeEnabled  : false

Active            : true
FCOENICName       : vmnic3
PrioritySettable  : false
SourceMAC         : fc:15:b4:15:b3:d5
SourceMACSettable : false
UserPriority      : 3
VLANRangeSettable : true
VN2VNModeEnabled  : false
tpcarman commented 5 years ago

Adding current PowerShell code for reference

Function Get-PciDeviceDetail {
        <#
    .SYNOPSIS
    Helper function to return PCI Devices Drivers & Firmware information for a specific host.
    .PARAMETER Server
    vCenter VISession object.
    .PARAMETER esxcli
    Esxcli session object associated to the host.
    .EXAMPLE
    $Credentials = Get-Crendentials
    $Server = Connect-VIServer -Server vcenter01.example.com -Credentials $Credentials
    $VMHost = Get-VMHost -Server $Server -Name esx01.example.com
    $esxcli = Get-EsxCli -Server $Server -VMHost $VMHost -V2
    Get-PciDeviceDetail -Server $vCenter -esxcli $esxcli
    VMkernel Name    : vmhba0
    Device Name      : Sunrise Point-LP AHCI Controller
    Driver           : vmw_ahci
    Driver Version   : 1.0.0-34vmw.650.0.14.5146846
    Firmware Version : NA
    VIB Name         : vmw-ahci
    VIB Version      : 1.0.0-34vmw.650.0.14.5146846
    .NOTES
    Author: Erwan Quelin heavily based on the work of the vDocumentation team - https://github.com/arielsanchezmora/vDocumentation/blob/master/powershell/vDocumentation/Public/Get-ESXIODevice.ps1
    #>
        [CmdletBinding()]
        Param (
            [Parameter(Mandatory = $true)]
            $Server,
            [Parameter(Mandatory = $true)]
            $esxcli
        )
        Begin { }

        Process {
            # Set default results
            $firmwareVersion = "N/A"
            $vibName = "N/A"
            $driverVib = @{
                Name = "N/A"
                Version = "N/A"
            }
            $pciDevices = $esxcli.hardware.pci.list.Invoke() | Where-Object { $_.VMKernelName -like "vmhba*" -or $_.VMKernelName -like "vmnic*" -or $_.VMKernelName -like "vmgfx*" } | Sort-Object -Property VMKernelName 
            foreach ($pciDevice in $pciDevices) {
                $driverVersion = $esxcli.system.module.get.Invoke(@{module = $pciDevice.ModuleName }) | Select-Object -ExpandProperty Version
                # Get NIC Firmware version
                if ($pciDevice.VMKernelName -like 'vmnic*') {
                    $vmnicDetail = $esxcli.network.nic.get.Invoke(@{nicname = $pciDevice.VMKernelName })
                    $firmwareVersion = $vmnicDetail.DriverInfo.FirmwareVersion
                    # Get NIC driver VIB package version
                    $driverVib = $esxcli.software.vib.list.Invoke() | Select-Object -Property Name, Version | Where-Object { $_.Name -eq $vmnicDetail.DriverInfo.Driver -or $_.Name -eq "net-" + $vmnicDetail.DriverInfo.Driver -or $_.Name -eq "net55-" + $vmnicDetail.DriverInfo.Driver }
                    <#
                    If HP Smart Array vmhba* (scsi-hpsa driver) then get Firmware version
                    else skip if VMkernnel is vmhba*. Can't get HBA Firmware from 
                    Powercli at the moment only through SSH or using Putty Plink+PowerCli.
                #>
                } elseif ($pciDevice.VMKernelName -like 'vmhba*') {
                    if ($pciDevice.DeviceName -match "smart array") {
                        $hpsa = $vmhost.ExtensionData.Runtime.HealthSystemRuntime.SystemHealthInfo.NumericSensorInfo | Where-Object { $_.Name -match "HP Smart Array" }
                        if ($hpsa) {
                            $firmwareVersion = (($hpsa.Name -split "firmware")[1]).Trim()
                        }
                    }
                    # Get HBA driver VIB package version
                    $vibName = $pciDevice.ModuleName -replace "_", "-"
                    $driverVib = $esxcli.software.vib.list.Invoke() | Select-Object -Property Name, Version | Where-Object { $_.Name -eq "scsi-" + $VibName -or $_.Name -eq "sata-" + $VibName -or $_.Name -eq $VibName }
                }
                # Output collected data
                [PSCustomObject]@{
                    'VMkernel Name' = $pciDevice.VMKernelName
                    'Device Name' = $pciDevice.DeviceName
                    'Driver' = $pciDevice.ModuleName
                    'Driver Version' = $driverVersion
                    'Firmware Version' = $firmwareVersion
                    'VIB Name' = $driverVib.Name
                    'VIB Version' = $driverVib.Version
                } 
            } 
        }
        End { }

    }