search

Napsty / check_esxi_hardware

Monitoring Plugin to check the hardware of VMware ESXi servers.
https://www.claudiokuenzler.com/monitoring-plugins/check_esxi_hardware.php
69 stars 18 forks source link

CIM Server issues/timeouts with ESXi 6.7 #37

Closed Napsty closed 5 years ago

Napsty commented 5 years ago

CIM issues in ESXi 6.7

In the past weeks, several users have reported issues with check_esxi_hardware since ESXi version 6.7.

This ticket is here to retrieve more information to pinpoint the issue. So far multiple hardware vendors and server models are affected. Please comment on this issue if you experience similar issues.

As of today, June 12 2019, I have not heard any feedback from VMware and Dell. I reached out to both of them via Twitter.

Related tickets

31

34

Two issues depending on ESXi version and hardware

  1. ESXi 6.7 (both with and without update packs): CIM server seems to have stability issues on Supermicro hardware. Users reported in #34 manual restarts are required.
  2. ESXi 6.7 U2: CIM server doesn't seem to work at all on DELL EMC hardware, times out on all requests when "OpenManage" VIB is installed. -> Issue in Dell OpenManage. ESXi 6.7 U2 is not officially supported by OpenManage yet. Supposed to be fixed in Fall 2019.

Affected server vendors

Workarounds

Issue 1: Restart of CIM server using /sbin/services.sh restart Issue 2: Uninstalling the "Openmanage" Offline Bundle VIB helps, the CIM server and therefore the plugin works again afterwards.

esxcli software vib remove --vibname=OpenManage

bridrod commented 5 years ago

I have the same issue. Working fine from ESX v4.1 up to ESXi v6.7U1 (Dell PowerEdge M620, M630, R640, R720, R730, R740, R815, R840, R905). So, far the only models I was able to test under v6.7U2 were: Dell R640, M630 and both time out and stop at the line "Check classe OMC_SMASHFirmwareIdentity" when using verbose output. Here is a sample of the VIBs installed (notice I am also using Dell OpenManage (tried with OMSA v9.2 and v9.3) on the hosts affected:

esxcli software vib list

Dell M630:

Name                           Version                               Vendor   Acceptance Level  Install Date
-----------------------------  ------------------------------------  -------  ----------------  ------------
dell-shared-perc8             06.806.90.00-1OEM.650.0.0.4598673     Avago    VMwareCertified   2019-05-29
bnxtnet                        212.0.119.0-1OEM.670.0.0.8169922      BCM      VMwareCertified   2019-05-29
bnxtroce                       212.0.114.0-1OEM.670.0.0.8169922      BCM      VMwareCertified   2019-05-29
dell-configuration-vib         6.7-2A00                              DellEMC  PartnerSupported  2019-05-29
dellemc-osname-idrac           6.7-0A03                              DellEMC  PartnerSupported  2019-05-29
OpenManage                     9.3.0.ESXi670-3465                    Dell     PartnerSupported  2019-05-30
elxnet                         12.0.1108.0-1OEM.600.0.0.2768847      EMU      VMwareCertified   2018-08-03
lpfc                           12.0.257.5-1OEM.670.0.0.8169922       EMU      VMwareCertified   2019-05-29
i40en-ens                      1.1.3-1OEM.670.0.0.8169922            INT      VMwareCertified   2019-05-29
i40en                          1.7.11-1OEM.670.0.0.8169922           INT      VMwareCertified   2019-05-29
igbn                           1.4.7-1OEM.670.0.0.8169922            INT      VMwareCertified   2019-05-29
ixgben-ens                     1.1.3-1OEM.670.0.0.8169922            INT      VMwareCertified   2019-05-29
ixgben                         1.7.10-1OEM.670.0.0.8169922           INT      VMwareCertified   2019-05-29
net-i40e                       2.0.7-1OEM.600.0.0.2494585            INT      VMwareCertified   2018-08-03
net-igb                        5.3.3-1OEM.600.0.0.2494585            INT      VMwareCertified   2018-08-03
net-ixgbe                      4.5.3-1OEM.600.0.0.2494585            INT      VMwareCertified   2018-08-03
nmlx5-core                     4.17.13.8-1OEM.670.0.0.8169922        MEL      VMwareCertified   2019-05-29
nmlx5-rdma                     4.17.13.8-1OEM.670.0.0.8169922        MEL      VMwareCertified   2019-05-29
qcnic                          1.0.22.0-1OEM.670.0.0.8169922         QLC      VMwareCertified   2019-05-29
qedentv                        3.9.31.2-1OEM.670.0.0.8169922         QLC      VMwareCertified   2019-05-29
qedf                           1.2.61.0-1OEM.600.0.0.2768847         QLC      VMwareCertified   2019-05-29
qedrntv                        3.9.31.2-1OEM.670.0.0.8169922         QLC      VMwareCertified   2019-05-29
qfle3                          1.0.77.2-1OEM.670.0.0.8169922         QLC      VMwareCertified   2019-05-29
qfle3f                         1.0.63.0-1OEM.670.0.0.8169922         QLC      VMwareCertified   2019-05-29
qfle3i                         1.0.20.0-1OEM.670.0.0.8169922         QLC      VMwareCertified   2019-05-29
scsi-qedil                     1.2.13.1-1OEM.600.0.0.2494585         QLC      VMwareCertified   2019-05-29
ima-qla4xxx                    500.2.02.20-1vmw.0.5.021123           QLogic   VMwareCertified   2018-08-03
misc-cnic-register             1.713.60.v60.1-1OEM.600.0.0.2494585   QLogic   VMwareCertified   2018-08-03
net-bnx2                       2.2.6b.v60.2-1OEM.600.0.0.2494585     QLogic   VMwareCertified   2017-08-15
net-bnx2x                      2.713.60.v60.2-1OEM.600.0.0.2494585   QLogic   VMwareCertified   2018-08-03
net-cnic                       2.713.60.v60.2-1OEM.600.0.0.2494585   QLogic   VMwareCertified   2018-08-03
net-qlcnic                     6.1.191-1OEM.600.0.0.2494585          QLogic   VMwareCertified   2016-10-07
net-qlge                       3.0.0.59-1OEM.550.0.0.1331820         QLogic   VMwareCertified   2016-10-07
qlnativefc                     3.1.16.0-1OEM.670.0.0.8169922         QLogic   VMwareCertified   2019-05-29
scsi-bnx2fc                    1.713.60.v60.1-1OEM.600.0.0.2494585   QLogic   VMwareCertified   2018-08-03
scsi-bnx2i                     2.713.60.v60.1-1OEM.600.0.0.2494585   QLogic   VMwareCertified   2018-08-03
scsi-qla4xxx                   644.6.06.0-1OEM.600.0.0.2494585       QLogic   VMwareCertified   2018-08-03
ata-libata-92                  3.00.9.2-16vmw.670.0.0.8169922        VMW      VMwareCertified   2019-05-29
ata-pata-amd                   0.3.10-3vmw.670.0.0.8169922           VMW      VMwareCertified   2019-05-29
ata-pata-atiixp                0.4.6-4vmw.670.0.0.8169922            VMW      VMwareCertified   2019-05-29
ata-pata-cmd64x                0.2.5-3vmw.670.0.0.8169922            VMW      VMwareCertified   2019-05-29
ata-pata-hpt3x2n               0.3.4-3vmw.670.0.0.8169922            VMW      VMwareCertified   2019-05-29
ata-pata-pdc2027x              1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
ata-pata-serverworks           0.4.3-3vmw.670.0.0.8169922            VMW      VMwareCertified   2019-05-29
ata-pata-sil680                0.4.8-3vmw.670.0.0.8169922            VMW      VMwareCertified   2019-05-29
ata-pata-via                   0.3.3-2vmw.670.0.0.8169922            VMW      VMwareCertified   2019-05-29
block-cciss                    3.6.14-10vmw.670.0.0.8169922          VMW      VMwareCertified   2019-05-29
brcmfcoe                       11.4.1078.19-12vmw.670.2.48.13006603  VMW      VMwareCertified   2019-05-29
char-random                    1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
ehci-ehci-hcd                  1.0-4vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
elxiscsi                       11.4.1174.0-2vmw.670.0.0.8169922      VMW      VMwareCertified   2019-05-29
hid-hid                        1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
iavmd                          1.2.0.1011-2vmw.670.0.0.8169922       VMW      VMwareCertified   2019-05-29
ipmi-ipmi-devintf              39.1-5vmw.670.1.28.10302608           VMW      VMwareCertified   2019-05-29
ipmi-ipmi-msghandler           39.1-5vmw.670.1.28.10302608           VMW      VMwareCertified   2019-05-29
ipmi-ipmi-si-drv               39.1-5vmw.670.1.28.10302608           VMW      VMwareCertified   2019-05-29
iser                           1.0.0.0-1vmw.670.1.28.10302608        VMW      VMwareCertified   2019-05-29
lpnic                          11.4.59.0-1vmw.670.0.0.8169922        VMW      VMwareCertified   2019-05-29
lsi-mr3                        7.708.07.00-2vmw.670.2.48.13006603    VMW      VMwareCertified   2019-05-29
lsi-msgpt2                     20.00.05.00-1vmw.670.2.48.13006603    VMW      VMwareCertified   2019-05-29
lsi-msgpt35                    09.00.00.00-1vmw.670.2.48.13006603    VMW      VMwareCertified   2019-05-29
lsi-msgpt3                     17.00.01.00-3vmw.670.2.48.13006603    VMW      VMwareCertified   2019-05-29
misc-drivers                   6.7.0-2.48.13006603                   VMW      VMwareCertified   2019-05-29
mtip32xx-native                3.9.8-1vmw.670.1.28.10302608          VMW      VMwareCertified   2019-05-29
ne1000                         0.8.4-2vmw.670.2.48.13006603          VMW      VMwareCertified   2019-05-29
nenic                          1.0.21.0-1vmw.670.1.28.10302608       VMW      VMwareCertified   2019-05-29
net-cdc-ether                  1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
net-e1000                      8.0.3.1-5vmw.670.0.0.8169922          VMW      VMwareCertified   2019-05-29
net-e1000e                     3.2.2.1-2vmw.670.0.0.8169922          VMW      VMwareCertified   2019-05-29
net-enic                       2.1.2.38-2vmw.670.0.0.8169922         VMW      VMwareCertified   2019-05-29
net-fcoe                       1.0.29.9.3-7vmw.670.0.0.8169922       VMW      VMwareCertified   2019-05-29
net-forcedeth                  0.61-2vmw.670.0.0.8169922             VMW      VMwareCertified   2019-05-29
net-libfcoe-92                 1.0.24.9.4-8vmw.670.0.0.8169922       VMW      VMwareCertified   2019-05-29
net-mlx4-core                  1.9.7.0-1vmw.670.0.0.8169922          VMW      VMwareCertified   2019-05-29
net-mlx4-en                    1.9.7.0-1vmw.670.0.0.8169922          VMW      VMwareCertified   2019-05-29
net-nx-nic                     5.0.621-5vmw.670.0.0.8169922          VMW      VMwareCertified   2019-05-29
net-tg3                        3.131d.v60.4-2vmw.670.0.0.8169922     VMW      VMwareCertified   2019-05-29
net-usbnet                     1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
net-vmxnet3                    1.1.3.0-3vmw.670.2.48.13006603        VMW      VMwareCertified   2019-05-29
nfnic                          4.0.0.17-0vmw.670.2.48.13006603       VMW      VMwareCertified   2019-05-29
nhpsa                          2.0.22-3vmw.670.1.28.10302608         VMW      VMwareCertified   2019-05-29
nmlx4-core                     3.17.13.1-1vmw.670.2.48.13006603      VMW      VMwareCertified   2019-05-29
nmlx4-en                       3.17.13.1-1vmw.670.2.48.13006603      VMW      VMwareCertified   2019-05-29
nmlx4-rdma                     3.17.13.1-1vmw.670.2.48.13006603      VMW      VMwareCertified   2019-05-29
ntg3                           4.1.3.2-1vmw.670.1.28.10302608        VMW      VMwareCertified   2019-05-29
nvme                           1.2.2.27-1vmw.670.2.48.13006603       VMW      VMwareCertified   2019-05-29
nvmxnet3-ens                   2.0.0.21-1vmw.670.0.0.8169922         VMW      VMwareCertified   2019-05-29
nvmxnet3                       2.0.0.29-1vmw.670.1.28.10302608       VMW      VMwareCertified   2019-05-29
ohci-usb-ohci                  1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
pvscsi                         0.1-2vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
qflge                          1.1.0.11-1vmw.670.0.0.8169922         VMW      VMwareCertified   2019-05-29
sata-ahci                      3.0-26vmw.670.0.0.8169922             VMW      VMwareCertified   2019-05-29
sata-ata-piix                  2.12-10vmw.670.0.0.8169922            VMW      VMwareCertified   2019-05-29
sata-sata-nv                   3.5-4vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
sata-sata-promise              2.12-3vmw.670.0.0.8169922             VMW      VMwareCertified   2019-05-29
sata-sata-sil24                1.1-1vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
sata-sata-sil                  2.3-4vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
sata-sata-svw                  2.3-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
scsi-aacraid                   1.1.5.1-9vmw.670.0.0.8169922          VMW      VMwareCertified   2019-05-29
scsi-adp94xx                   1.0.8.12-6vmw.670.0.0.8169922         VMW      VMwareCertified   2019-05-29
scsi-aic79xx                   3.1-6vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
scsi-fnic                      1.5.0.45-3vmw.670.0.0.8169922         VMW      VMwareCertified   2019-05-29
scsi-hpsa                      6.0.0.84-3vmw.670.0.0.8169922         VMW      VMwareCertified   2019-05-29
scsi-ips                       7.12.05-4vmw.670.0.0.8169922          VMW      VMwareCertified   2019-05-29
scsi-iscsi-linux-92            1.0.0.2-3vmw.670.0.0.8169922          VMW      VMwareCertified   2019-05-29
scsi-libfc-92                  1.0.40.9.3-5vmw.670.0.0.8169922       VMW      VMwareCertified   2019-05-29
scsi-megaraid-mbox             2.20.5.1-6vmw.670.0.0.8169922         VMW      VMwareCertified   2019-05-29
scsi-megaraid-sas              06.805.56.00-1OEM.600.0.0.2494585     VMW      VMwareCertified   2018-08-03
scsi-megaraid2                 2.00.4-9vmw.670.0.0.8169922           VMW      VMwareCertified   2019-05-29
scsi-mpt2sas                   19.00.00.00-2vmw.670.0.0.8169922      VMW      VMwareCertified   2019-05-29
scsi-mptsas                    4.23.01.00-10vmw.670.0.0.8169922      VMW      VMwareCertified   2019-05-29
scsi-mptspi                    4.23.01.00-10vmw.670.0.0.8169922      VMW      VMwareCertified   2019-05-29
sfvmk                          1.0.0.1003-6vmw.670.2.48.13006603     VMW      VMwareCertified   2019-05-29
shim-iscsi-linux-9-2-1-0       6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-iscsi-linux-9-2-2-0       6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-libata-9-2-1-0            6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-libata-9-2-2-0            6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-libfc-9-2-1-0             6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-libfc-9-2-2-0             6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-libfcoe-9-2-1-0           6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-libfcoe-9-2-2-0           6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-vmklinux-9-2-1-0          6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-vmklinux-9-2-2-0          6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
shim-vmklinux-9-2-3-0          6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
smartpqi                       1.0.1.553-12vmw.670.1.28.10302608     VMW      VMwareCertified   2019-05-29
uhci-usb-uhci                  1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
usb-storage-usb-storage        1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
usbcore-usb                    1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
vmkata                         0.1-1vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
vmkfcoe                        1.0.0.1-1vmw.670.1.28.10302608        VMW      VMwareCertified   2019-05-29
vmkplexer-vmkplexer            6.7.0-0.0.8169922                     VMW      VMwareCertified   2019-05-29
vmkusb                         0.1-1vmw.670.2.48.13006603            VMW      VMwareCertified   2019-05-29
vmw-ahci                       1.2.3-1vmw.670.1.28.10302608          VMW      VMwareCertified   2019-05-29
xhci-xhci                      1.0-3vmw.670.0.0.8169922              VMW      VMwareCertified   2019-05-29
cpu-microcode                  6.7.0-1.28.10302608                   VMware   VMwareCertified   2019-05-29
elx-esx-libelxima.so           11.4.1184.1-2.48.13006603             VMware   VMwareCertified   2019-05-29
epsec-mux                      6.5.0esx65-12742868                   VMware   VMwareCertified   2019-05-30
esx-base                       6.7.0-2.48.13006603                   VMware   VMwareCertified   2019-05-29
esx-dvfilter-generic-fastpath  6.7.0-0.0.8169922                     VMware   VMwareCertified   2019-05-29
esx-ui                         1.33.3-12923304                       VMware   VMwareCertified   2019-05-29
esx-update                     6.7.0-2.48.13006603                   VMware   VMwareCertified   2019-05-29
esx-xserver                    6.7.0-0.0.8169922                     VMware   VMwareCertified   2019-05-29
lsu-hp-hpsa-plugin             2.0.0-16vmw.670.1.28.10302608         VMware   VMwareCertified   2019-05-29
lsu-intel-vmd-plugin           1.0.0-2vmw.670.1.28.10302608          VMware   VMwareCertified   2019-05-29
lsu-lsi-drivers-plugin         1.0.0-1vmw.670.2.48.13006603          VMware   VMwareCertified   2019-05-29
lsu-lsi-lsi-mr3-plugin         1.0.0-13vmw.670.1.28.10302608         VMware   VMwareCertified   2019-05-29
lsu-lsi-lsi-msgpt3-plugin      1.0.0-9vmw.670.2.48.13006603          VMware   VMwareCertified   2019-05-29
lsu-lsi-megaraid-sas-plugin    1.0.0-9vmw.670.0.0.8169922            VMware   VMwareCertified   2019-05-29
lsu-lsi-mpt2sas-plugin         2.0.0-7vmw.670.0.0.8169922            VMware   VMwareCertified   2019-05-29
lsu-smartpqi-plugin            1.0.0-3vmw.670.1.28.10302608          VMware   VMwareCertified   2019-05-29
native-misc-drivers            6.7.0-2.48.13006603                   VMware   VMwareCertified   2019-05-29
rste                           2.0.2.0088-7vmw.670.0.0.8169922       VMware   VMwareCertified   2019-05-29
vmware-esx-esxcli-nvme-plugin  1.2.0.36-2.48.13006603                VMware   VMwareCertified   2019-05-29
vmware-fdm                     6.5.0-8815520                         VMware   VMwareCertified   2018-07-31
vsan                           6.7.0-2.48.12775451                   VMware   VMwareCertified   2019-05-29
vsanhealth                     6.7.0-2.48.12775454                   VMware   VMwareCertified   2019-05-29
tools-light                    10.3.5.10430147-12986307              VMware   VMwareCertified   2019-05-29

Dell R640:

Name                           Version                                 Vendor   Acceptance Level  Install Date
-----------------------------  --------------------------------------  -------  ----------------  ------------
dell-shared-perc8              06.806.90.00-1OEM.650.0.0.4598673       Avago    VMwareCertified   2019-01-24
lsi-mr3                        7.705.10.00-1OEM.670.0.0.8169922        Avago    VMwareCertified   2019-01-24
lsi-msgpt3                     17.00.01.00-1OEM.670.0.0.8169922        Avago    VMwareCertified   2019-01-24
bnxtnet                        212.0.119.0-1OEM.670.0.0.8169922        BCM      VMwareCertified   2019-01-24
bnxtroce                       212.0.114.0-1OEM.670.0.0.8169922        BCM      VMwareCertified   2019-01-24
dell-configuration-vib         6.7-1A02                                DellEMC  PartnerSupported  2019-01-24
dellemc-osname-idrac           6.7-0A01                                DellEMC  PartnerSupported  2019-01-24
OpenManage                     9.2.0.ESXi670-3212                      Dell     PartnerSupported  2019-02-06
lpfc                           12.0.257.5-1OEM.670.0.0.8169922         EMU      VMwareCertified   2019-01-24
i40en-ens                      1.1.3-1OEM.670.0.0.8169922              INT      VMwareCertified   2019-01-24
i40en                          1.7.11-1OEM.670.0.0.8169922             INT      VMwareCertified   2019-01-24
igbn                           1.4.7-1OEM.670.0.0.8169922              INT      VMwareCertified   2019-01-24
ixgben-ens                     1.1.3-1OEM.670.0.0.8169922              INT      VMwareCertified   2019-01-24
ixgben                         1.7.10-1OEM.670.0.0.8169922             INT      VMwareCertified   2019-01-24
nmlx5-core                     4.17.13.8-1OEM.670.0.0.8169922          MEL      VMwareCertified   2019-01-24
nmlx5-rdma                     4.17.13.8-1OEM.670.0.0.8169922          MEL      VMwareCertified   2019-01-24
qcnic                          1.0.16.0-1OEM.670.0.0.7535516           QLC      VMwareCertified   2019-01-24
qedentv                        3.7.9.1-1OEM.670.0.0.7535516            QLC      VMwareCertified   2019-01-24
qedf                           1.2.24.6-1OEM.600.0.0.2768847           QLC      VMwareCertified   2019-01-24
qedrntv                        3.7.9.2-1OEM.670.0.0.7535516            QLC      VMwareCertified   2019-01-24
qfle3                          1.0.69.0-1OEM.670.0.0.7535516           QLC      VMwareCertified   2019-01-24
qfle3f                         1.0.51.0-1OEM.670.0.0.7535516           QLC      VMwareCertified   2019-01-24
qfle3i                         1.0.16.0-1OEM.670.0.0.7535516           QLC      VMwareCertified   2019-01-24
scsi-qedil                     1.0.22.0-1OEM.600.0.0.2494585           QLC      VMwareCertified   2019-01-24
qlnativefc                     3.1.9.0-1OEM.670.0.0.8169922            QLogic   VMwareCertified   2019-01-24
ata-libata-92                  3.00.9.2-16vmw.670.1.28.10302608        VMW      VMwareCertified   2019-01-24
ata-pata-amd                   0.3.10-3vmw.670.1.28.10302608           VMW      VMwareCertified   2019-01-24
ata-pata-atiixp                0.4.6-4vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
ata-pata-cmd64x                0.2.5-3vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
ata-pata-hpt3x2n               0.3.4-3vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
ata-pata-pdc2027x              1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
ata-pata-serverworks           0.4.3-3vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
ata-pata-sil680                0.4.8-3vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
ata-pata-via                   0.3.3-2vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
block-cciss                    3.6.14-10vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
brcmfcoe                       11.4.1078.5-11vmw.670.1.28.10302608     VMW      VMwareCertified   2019-01-24
char-random                    1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
ehci-ehci-hcd                  1.0-4vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
elxiscsi                       11.4.1174.0-2vmw.670.1.28.10302608      VMW      VMwareCertified   2019-01-24
elxnet                         11.4.1095.0-5vmw.670.1.28.10302608      VMW      VMwareCertified   2019-01-24
hid-hid                        1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
iavmd                          1.2.0.1011-2vmw.670.1.28.10302608       VMW      VMwareCertified   2019-01-24
ima-qla4xxx                    2.02.18-1vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
ipmi-ipmi-devintf              39.1-5vmw.670.1.28.10302608             VMW      VMwareCertified   2019-01-24
ipmi-ipmi-msghandler           39.1-5vmw.670.1.28.10302608             VMW      VMwareCertified   2019-01-24
ipmi-ipmi-si-drv               39.1-5vmw.670.1.28.10302608             VMW      VMwareCertified   2019-01-24
iser                           1.0.0.0-1vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
lpnic                          11.4.60.0-1vmw.670.1.28.10302608        VMW      VMwareCertified   2019-01-24
lsi-msgpt2                     20.00.04.00-5vmw.670.1.28.10302608      VMW      VMwareCertified   2019-01-24
lsi-msgpt35                    03.00.01.00-12vmw.670.1.28.10302608     VMW      VMwareCertified   2019-01-24
misc-cnic-register             1.78.75.v60.7-1vmw.670.1.28.10302608    VMW      VMwareCertified   2019-01-24
misc-drivers                   6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
mtip32xx-native                3.9.8-1vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
ne1000                         0.8.4-1vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
nenic                          1.0.21.0-1vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
net-bnx2                       2.2.4f.v60.10-2vmw.670.1.28.10302608    VMW      VMwareCertified   2019-01-24
net-bnx2x                      1.78.80.v60.12-2vmw.670.1.28.10302608   VMW      VMwareCertified   2019-01-24
net-cdc-ether                  1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
net-cnic                       1.78.76.v60.13-2vmw.670.1.28.10302608   VMW      VMwareCertified   2019-01-24
net-e1000                      8.0.3.1-5vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
net-e1000e                     3.2.2.1-2vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
net-enic                       2.1.2.38-2vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
net-fcoe                       1.0.29.9.3-7vmw.670.1.28.10302608       VMW      VMwareCertified   2019-01-24
net-forcedeth                  0.61-2vmw.670.1.28.10302608             VMW      VMwareCertified   2019-01-24
net-igb                        5.0.5.1.1-5vmw.670.1.28.10302608        VMW      VMwareCertified   2019-01-24
net-ixgbe                      3.7.13.7.14iov-20vmw.670.1.28.10302608  VMW      VMwareCertified   2019-01-24
net-libfcoe-92                 1.0.24.9.4-8vmw.670.1.28.10302608       VMW      VMwareCertified   2019-01-24
net-mlx4-core                  1.9.7.0-1vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
net-mlx4-en                    1.9.7.0-1vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
net-nx-nic                     5.0.621-5vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
net-tg3                        3.131d.v60.4-2vmw.670.1.28.10302608     VMW      VMwareCertified   2019-01-24
net-usbnet                     1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
net-vmxnet3                    1.1.3.0-3vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
nfnic                          4.0.0.14-0vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
nhpsa                          2.0.22-3vmw.670.1.28.10302608           VMW      VMwareCertified   2019-01-24
nmlx4-core                     3.17.9.12-1vmw.670.1.28.10302608        VMW      VMwareCertified   2019-01-24
nmlx4-en                       3.17.9.12-1vmw.670.1.28.10302608        VMW      VMwareCertified   2019-01-24
nmlx4-rdma                     3.17.9.12-1vmw.670.1.28.10302608        VMW      VMwareCertified   2019-01-24
ntg3                           4.1.3.2-1vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
nvme                           1.2.2.17-1vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
nvmxnet3-ens                   2.0.0.21-1vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
nvmxnet3                       2.0.0.29-1vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
ohci-usb-ohci                  1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
pvscsi                         0.1-2vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
qflge                          1.1.0.11-1vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
sata-ahci                      3.0-26vmw.670.1.28.10302608             VMW      VMwareCertified   2019-01-24
sata-ata-piix                  2.12-10vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
sata-sata-nv                   3.5-4vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
sata-sata-promise              2.12-3vmw.670.1.28.10302608             VMW      VMwareCertified   2019-01-24
sata-sata-sil24                1.1-1vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
sata-sata-sil                  2.3-4vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
sata-sata-svw                  2.3-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
scsi-aacraid                   1.1.5.1-9vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
scsi-adp94xx                   1.0.8.12-6vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
scsi-aic79xx                   3.1-6vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
scsi-bnx2fc                    1.78.78.v60.8-1vmw.670.1.28.10302608    VMW      VMwareCertified   2019-01-24
scsi-bnx2i                     2.78.76.v60.8-1vmw.670.1.28.10302608    VMW      VMwareCertified   2019-01-24
scsi-fnic                      1.5.0.45-3vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
scsi-hpsa                      6.0.0.84-3vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
scsi-ips                       7.12.05-4vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
scsi-iscsi-linux-92            1.0.0.2-3vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
scsi-libfc-92                  1.0.40.9.3-5vmw.670.1.28.10302608       VMW      VMwareCertified   2019-01-24
scsi-megaraid-mbox             2.20.5.1-6vmw.670.1.28.10302608         VMW      VMwareCertified   2019-01-24
scsi-megaraid-sas              6.603.55.00-2vmw.670.1.28.10302608      VMW      VMwareCertified   2019-01-24
scsi-megaraid2                 2.00.4-9vmw.670.1.28.10302608           VMW      VMwareCertified   2019-01-24
scsi-mpt2sas                   19.00.00.00-2vmw.670.1.28.10302608      VMW      VMwareCertified   2019-01-24
scsi-mptsas                    4.23.01.00-10vmw.670.1.28.10302608      VMW      VMwareCertified   2019-01-24
scsi-mptspi                    4.23.01.00-10vmw.670.1.28.10302608      VMW      VMwareCertified   2019-01-24
scsi-qla4xxx                   5.01.03.2-7vmw.670.1.28.10302608        VMW      VMwareCertified   2019-01-24
shim-iscsi-linux-9-2-1-0       6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-iscsi-linux-9-2-2-0       6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-libata-9-2-1-0            6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-libata-9-2-2-0            6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-libfc-9-2-1-0             6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-libfc-9-2-2-0             6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-libfcoe-9-2-1-0           6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-libfcoe-9-2-2-0           6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-vmklinux-9-2-1-0          6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-vmklinux-9-2-2-0          6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
shim-vmklinux-9-2-3-0          6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
smartpqi                       1.0.1.553-12vmw.670.1.28.10302608       VMW      VMwareCertified   2019-01-24
uhci-usb-uhci                  1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
usb-storage-usb-storage        1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
usbcore-usb                    1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
vmkata                         0.1-1vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
vmkfcoe                        1.0.0.1-1vmw.670.1.28.10302608          VMW      VMwareCertified   2019-01-24
vmkplexer-vmkplexer            6.7.0-1.28.10302608                     VMW      VMwareCertified   2019-01-24
vmkusb                         0.1-1vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
vmw-ahci                       1.2.3-1vmw.670.1.28.10302608            VMW      VMwareCertified   2019-01-24
xhci-xhci                      1.0-3vmw.670.1.28.10302608              VMW      VMwareCertified   2019-01-24
cpu-microcode                  6.7.0-1.28.10302608                     VMware   VMwareCertified   2019-01-24
elx-esx-libelxima.so           11.4.1184.0-1.28.10302608               VMware   VMwareCertified   2019-01-24
epsec-mux                      6.5.0esx65-9573428                      VMware   VMwareCertified   2019-03-05
esx-base                       6.7.0-1.31.10764712                     VMware   VMwareCertified   2019-02-12
esx-dvfilter-generic-fastpath  6.7.0-1.28.10302608                     VMware   VMwareCertified   2019-01-24
esx-ui                         1.30.0-9946814                          VMware   VMwareCertified   2019-01-24
esx-update                     6.7.0-1.31.10764712                     VMware   VMwareCertified   2019-02-12
esx-xserver                    6.7.0-1.28.10302608                     VMware   VMwareCertified   2019-01-24
lsu-hp-hpsa-plugin             2.0.0-16vmw.670.1.28.10302608           VMware   VMwareCertified   2019-01-24
lsu-intel-vmd-plugin           1.0.0-2vmw.670.1.28.10302608            VMware   VMwareCertified   2019-01-24
lsu-lsi-lsi-mr3-plugin         1.0.0-13vmw.670.1.28.10302608           VMware   VMwareCertified   2019-01-24
lsu-lsi-lsi-msgpt3-plugin      1.0.0-8vmw.670.1.28.10302608            VMware   VMwareCertified   2019-01-24
lsu-lsi-megaraid-sas-plugin    1.0.0-9vmw.670.1.28.10302608            VMware   VMwareCertified   2019-01-24
lsu-lsi-mpt2sas-plugin         2.0.0-7vmw.670.1.28.10302608            VMware   VMwareCertified   2019-01-24
lsu-smartpqi-plugin            1.0.0-3vmw.670.1.28.10302608            VMware   VMwareCertified   2019-01-24
native-misc-drivers            6.7.0-1.28.10302608                     VMware   VMwareCertified   2019-01-24
rste                           2.0.2.0088-7vmw.670.1.28.10302608       VMware   VMwareCertified   2019-01-24
vmware-esx-esxcli-nvme-plugin  1.2.0.34-1.28.10302608                  VMware   VMwareCertified   2019-01-24
vmware-fdm                     6.7.0-10244745                          VMware   VMwareCertified   2019-02-19
vsan                           6.7.0-1.31.10720746                     VMware   VMwareCertified   2019-02-12
vsanhealth                     6.7.0-1.31.10720754                     VMware   VMwareCertified   2019-02-12
tools-light                    10.3.2.9925305-10302608                 VMware   VMwareCertified   2019-01-24

Hope this will help you folks figure it out.

Edit by @Napsty : Adjusted format for better readability

philrandal commented 5 years ago

I upgraded a couple of ESXi 6.7U1 Dell R740 hosts with BIOS and iDRAC firmware up to date running Dell OpenManage 9.2 to 6.7U2 plus latest security update and got timeouts querying CIM.

Updated to OpenManage 9.3 without any change.

Uninstalling OpenManage fixed it.

philrandal commented 5 years ago

There's a discussion over at Spiceworks:

https://community.spiceworks.com/topic/2213257-dell-omsa-9-2-esxi-6-7u2

philrandal commented 5 years ago

Dell's OpenManage Integration for VMware vCenter doesn't support 6.7U2.

Their web page (https://www.dell.com/support/article/us/en/04/sln311238/openmanage-integration-for-vmware-vcenter) says:

"Does not add official 6.7 U2 support (support for 6.7 U2 will come in the fall with the next major release)"

So we could be waiting a while for an updated OpenManage vib

Napsty commented 5 years ago

Thanks @philrandal for your research!

So this would explain the issues for Dell servers with ESXi 6.7 U2, but in the other issue Supermicro was also mentioned (by @MarcusCaepio) . We need a confirmation/verification that it's only Dell's OpenManage or the other way around, that other vendors are also affected.

In the meantime I have created a support ticket at VMware to get a statement.

ucola commented 5 years ago

can confirm that Supermicro also have this issue

Napsty commented 5 years ago

@ucola Let's assume there are two different issues:

  1. General stability issues of the CIM server in ESXi 6.7. Sometimes it works (after sbcfd-watchdog restart).
  2. CIM timeouts on ESXi 6.7 U2. Nothing works here, a restart of the server or the sbcfd-watchdog has no impact.

Are the Supermicro servers only affected by 1 or 2 or both?

ucola commented 5 years ago

thank you for your answer, Supermicro with ESXi 6.7 U1 gives me after some time the Timeout, but after restarts (/sbin/services.sh restart) it works again for a while.

Napsty commented 5 years ago

@ucola OK that looks like the first issue applies to Supermicro. It would be interesting to see how it behaves with ESXi 6.7 U2.

Napsty commented 5 years ago

Could someone please check how the issues behave in vSphere Web Client when accessing the server's hardware tab? Does this timeout as well (with the issue 2/ESXi 6.7 U2 with OpenManage)?

image

Napsty commented 5 years ago

@ucola Do you have to /sbin/services.sh restartor is a restart of the sfcbd-watchdog enough? -> /etc/init.d/sfcbd-watchdog restart

bridrod commented 5 years ago

Could someone please check how the issues behave in vSphere Web Client when accessing the server's hardware tab? Does this timeout as well (with the issue 2/ESXi 6.7 U2 with OpenManage)?

image

I compared to another R640 with v6.7.0 build=10764712 (before U2) and the output is the same. Some sensors show unknown but most show status=normal comparing to R640 with v6.7.0 build=13006603 (U2).

Napsty commented 5 years ago

@bridrod So "Hardware Health" is shown correctly and without timeouts (when you hit "Refresh") on a Dell server with ESXi 6.7 U2 and OpenManage VIB installed?

ucola commented 5 years ago

@Napsty /sbin/services.sh restart works fine... i put this on a cronjob every hour...

bridrod commented 5 years ago

@bridrod So "Hardware Health" is shown correctly and without timeouts (when you hit "Refresh") on a Dell server with ESXi 6.7 U2 and OpenManage VIB installed?

I refreshed on both versions and they both took the tasks and came out as completed. Interesting the last updated column did not change though. Not sure how it should behave. I actually never used the hardware health tab before. Relied only on either Dell Open Manage Essentials and Enterprise appliance (to monitor Dell hardware) and the check_esxi_hardware script.

Napsty commented 5 years ago

A colleague of mine added two new ESXi 6.7 U2 servers into the datacenter, so I can test locally now too. I immediately hit the problem reported by @ucola with the error Socket error: [Errno 104] Connection reset by peer'. However this disappeared immediately after applying https://kb.vmware.com/s/article/1025757 and the CIM server responded, plugin therefore worked.

I will keep an eye on this in the next days and see if there are stability issues.

This is on Cisco UCSB-B200-M5 by the way.

ucola commented 5 years ago

@Napsty Now my ESXi is updated to 6.7 U2, same issue "<Terminated by signal 9 (Killed).>"

hampe4460 commented 5 years ago

I have the same issue on HP DL380 G8, ESXi 6.7U2 when I check with Icinga2 --> <Terminated by signal 9 (Killed).> Local check from console works fine. HP DL380 G7 works for both, console and Icinga2

Napsty commented 5 years ago

@hampe4460 Can you please elaborate what you mean with "console" ?

hampe4460 commented 5 years ago

Sorry for the "console", I mean Debian CLI /usr/lib/nagios/plugins# ./check_esxi_hardware.py -H myIP -U file:/myfile -P file:/myfile OK - Server: HP ProLiant DL380p Gen8 s/n: CZ5671FR9K System BIOS: P70 2014-08-02

Napsty commented 5 years ago

@hampe4460 Thanks. Can you measure how long it takes? time ./check_esxi_hardware.py -H myIP -U file:/myfile -P file:/myfile. @ucola Does your error <Terminated by signal 9 (Killed).> only show up in Icinga 2 or also when you run the plugin manually in a terminal?

ucola commented 5 years ago

@Napsty also in the terminal...

hampe4460 commented 5 years ago

It takes around 12 seconds, and it works every time I try it. The error message <Terminated by signal 9 (Killed).> is only in Icinga2. I tried it there with the -t switch to set it to 60, but this didn't work.

Napsty commented 5 years ago

@hampe4460 You might need to increase the command timeout in Icinga2:

object CheckCommand "esxi_hardware" {
  import "plugin-check-command"

  command = [ PluginContribDir + "/check_esxi_hardware.py" ]
   timeout = 180

  arguments = {
    "-H" = {
      value = "$esxi_hardware_host$"
      description = "report on HOS
[...]

But be aware that if you change this in ITL (/usr/share/icinga2/include/plugins-contrib.d/virtualization.conf) it might be overwritten again after an update. So you could just copy the whole command definition from /usr/share/icinga2/include/plugins-contrib.d/virtualization.conf and create a new command, e.g. check_esxi_hardware with a check prefix. Also see https://www.claudiokuenzler.com/monitoring-plugins/check_esxi_hardware.php#CommandDefinition.

@ucola Can you run the plugin on the terminal in verbose mode (-v) with time?

hampe4460 commented 5 years ago

I added the timeout = 180 option to /usr/share/icinga2/include/plugins-contrib.d/virtualization.conf, still the same

Napsty commented 5 years ago

@hampe4460 You reloaded icinga2 afterwards, right? Maybe this timeout is overwritten by something else, see https://github.com/Icinga/icinga2/issues/6992 . As you can run the plugin fine on the console/terminal you might have to check with the Icinga people why and where the CheckCommand's timeout is not respected.

hampe4460 commented 5 years ago

Yes, I rebootet the whole server. I also was looking for a solution on the Icinga2 side, but no luck. I don't know Icinga 2 good enough to figure out the right switches to set correctly...

hampe4460 commented 5 years ago

You were so right, I just figured out what I needed to change, was too easy at the end... hopefully. Needed to create a new service template and there I extended the timeout to 120, and now it works... Thank you all for your great help!

Napsty commented 5 years ago

FYI I opened a VMware support ticket concerning the CIM server issues. However I and VMware are unable to reproduce. The issues with the OpenManage VIB are obviously caused by Dell's OpenManage and that's not VMware's problem.

@ucola Is there a way we can have a remote session to try and debug this?

ucola commented 5 years ago

@Napsty of corse, you can contact me...

briancamp commented 5 years ago

FYI, from Dell -

We have an update; none of our versions of 6.7u2 doesn't support any of our current versions of OMSA.

There will be a supported version 9.3.1 that is still in development - no ETA was given, other than it's road mapped for later this year. This is likely as well why the hostname is not properly populating in the iDrac.

We wouldn't recommend that you downgrade; rather, our recommendation would be to manage the host via the iDrac.

Napsty commented 5 years ago

To close this issue, which are actually two separate cases (see updated description at the begin): 1) All ESXi 6.7 versions and affects only Supermicro hardware (not sure if all models or only certain models though). Unfortunately VMware was unable to help identify the reason for the CIM server instabilities. I created a VMware support case but it was not resolved. 2) ESXi 6.7 U2 affects only DELL hardware when the "openmanage" VIB is installed. Openmanage is not yet officially supported on ESXi 6.7 U2. A fix is supposed to be released in fall 2019.

Both cases are caused by third party tools. Nothing where the check_esxi_hardware could do something against. Thanks to everyone involved in troubleshooting and giving information. It is highly appreciated. Bummer that VMware wouldn't give more support and effort into this.

philrandal commented 5 years ago

VMware's KB article - ESXi 6.7 U2/U3 unresponsive when running Dell OpenManage Server Administrator 9.3.0:

https://kb.vmware.com/s/article/74696

bridrod commented 4 years ago

Yep. The solution was to remove OpenManage indeed. In our case, we were not gaining much anyway, by having it installed.

sittem commented 4 years ago

Has anyone tried new OMSA Version 9.3.1 (OM-SrvAdmin-Dell-Web-9.3.1-3684.VIB-ESX67i_A00.zip)? https://www.dell.com/support/article/au/en/aubsd1/sln318662/omsa-hosts-with-esxi-6-7-update-3-become-unresponsive-to-vcenter-with-task-time-outs-if-omsa-9-3-0-installed?lang=en

sittem commented 4 years ago

OMSA 9.3.1 has fixed the problem