Connection error after ESXi upgrade

[fvb-sb]

Dear community

After upgrading my ESXi hosts from 6.7.0, 17700523 to 7.0.3, 18644231 the check_esxi_hardware script doesn't work anymore. CLI output is:

20211027 15:21:47 LCD Status: True 20211027 15:21:47 Chassis Intrusion Status: True 20211027 15:21:47 Connection to https://xxx.xxx.xx 20211027 15:21:47 Found pywbem version 0.7.0 20211027 15:21:47 Connection error, disable SSL certificate verification (probably patched pywbem) 20211027 15:21:47 Check classe OMC_SMASHFirmwareIdentity Traceback (most recent call last): File "./check_esxi_hardware.py", line 790, in except pywbem._exceptions.ConnectionError as args: AttributeError: 'module' object has no attribute '_exceptions'

I've tried updating pywbem as well. Since then CLI output has changed but still no success:

20211027 15:27:35 LCD Status: True 20211027 15:27:35 Chassis Intrusion Status: True 20211027 15:27:35 Connection to https://xxx.xxx.xx 20211027 15:27:35 Found pywbem version 1.2.0 20211027 15:27:35 Check classe OMC_SMASHFirmwareIdentity Traceback (most recent call last): File "./check_esxi_hardware.py", line 780, in instance_list = wbemclient.EnumerateInstances(classe) File "/usr/local/lib/python3.6/site-packages/pywbem/_cim_operations.py", line 2783, in EnumerateInstances PropertyList=PropertyList) File "/usr/local/lib/python3.6/site-packages/pywbem/_cim_operations.py", line 1824, in _imethodcall self, request_data, cimxml_headers) File "/usr/local/lib/python3.6/site-packages/pywbem/_cim_http.py", line 389, in wbem_request raise AuthError(msg, conn_id=conn.conn_id) pywbem._exceptions.AuthError: WBEM server returned HTTP status 401 (Unauthorized). This is most likely an issue with userid/password, but for servers that implement resource access control it might also be an issue with the permissions of the userid.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "./check_esxi_hardware.py", line 781, in except pywbem.cim_operations.CIMError as args: AttributeError: module 'pywbem' has no attribute 'cim_operations'

I've one ESXi node left running on 6.7. For this node the script still works like a charm. Any ideas what could do the trick?

OS: CentOS7 pywbem: 0.7.0 / 1.2.0 Version of check_esxi_hardware.py: 20210809 Executed command: check_esxi_hardware.py -H HOSTNAME -U USERNAME -P PASSWORD -V dell -v

Any help appreciated. Thanks in advance. Sebastian

[Napsty]

This sounds like a DELL OMSA issue. Please check the FAQ: https://www.claudiokuenzler.com/blog/308/check-esxi-hardware-faq-frequently-asked-questions. You need to upgrade the OpenManage VIB.

[Napsty]

Sorry, I may have closed the issue too quickly.

pywbem._exceptions.AuthError: WBEM server returned HTTP status 401 (Unauthorized). This is most likely an issue with userid/password, but for servers that implement resource access control it might also be an issue with the permissions of the userid.

So this would indicate wrong credentials. But nevertheless, check the OpenManage status.

[fvb-sb]

Thanks for your swift reply Claudio.

I've checked the installed VIBs on the 7.0.3 and 6.7 hosts. It seems that there are no OpenManage VIBs at all:

7.0.3:

esxcli software vib list | grep -iE 'del|open|manag' dell-shared-perc8 06.806.92.00-1OEM.700.1.0.15843807 BCM VMwareCertified 2021-10-14 dell-configuration-vib 7.0.0-A00 DEL PartnerSupported 2021-10-14 dellemc-osname-idrac 7.0.0-A00 DEL PartnerSupported 2021-10-14 lsuv2-oem-dell-plugin 1.0.0-1vmw.703.0.0.18644231 VMware VMwareCertified 2021-10-14

6.7:

esxcli software vib list | grep -iE 'del|open|manag' dell-shared-perc8 06.806.90.00-1OEM.650.0.0.4598673 Avago VMwareCertified 2018-12-28 dell-configuration-vib 6.7-1A02 DellEMC PartnerSupported 2018-12-28 dellemc-osname-idrac 6.7-0A01 DellEMC PartnerSupported 2018-12-28

The credentials seem to be fine - the user is still present on the affected ESXi hosts and login through WebUI works well.

For the ESXi upgrade I've used a custom OEM ISO (downloaded on my.vmware.com: VMware-VMvisor-Installer-7.0.0.update03-18644231.x86_64-DellEMC_Customized-A00.iso) - may this be a problem when using this instead of the 'pure' ISO provided by VMware?

Kind regards Sebastian

[Napsty]

Strange, I always thought the Dell Customized ISO contains the OpenManage VIB. I suggest to try two things:

Manually download and install the OpenManage VIB on your already installed Dell custom ESXi. If this doesn't work, install the normal ESXi and see if the CIM elements works (you can use the plugin to test this).

On Dell servers the OpenManage VIB is recommended, as it adds additional CIM/hardware elements to the list.

Yes, try it with the normal ESXi ISO.

[fvb-sb]

It turned out that the permission of the monitoring user on the ESXi hosts were changed somehow due to the upgrade from ESXi 6.7 to 7.0. After applying administrative rights to the monitoring user the check_esxi_hardware.py script works as before/expected.

[lasag-2511]

Hi is there any way that we check with no ssl certificate

[Napsty]

tcp/5989 is the CIM server with SSL/TLS certificate (wbem-https). The alternative is tcp/5988 which is the CIM server without encryption (wbem-http). But I'm not sure if this is enabled or can be enabled on ESXi side. And even if it were possible, I strongly advise against using it, as your login credentials would be transmitted in plain text.

[lasag-2511]

yes i disabled on esxi CIM secure server, i.e 5989, but still i am having issues UNKNOWN: 'Connection to "IP" timed out. (connect timeout=9.99)'

^CTraceback (most recent call last): File "/usr/lib/nagios/plugins/check_esxi.py", line 804, in instance_list = wbemclient.EnumerateInstances(classe) File "/usr/local/lib/python2.7/dist-packages/pywbem/_cim_operations.py", line 3060, in EnumerateInstances PropertyList=PropertyList) File "/usr/local/lib/python2.7/dist-packages/pywbem/_cim_operations.py", line 1950, in _imethodcall self, request_data, cimxml_headers) File "/usr/local/lib/python2.7/dist-packages/pywbem/_cim_http.py", line 494, in wbem_request timeout=(HTTP_CONNECT_TIMEOUT, conn.timeout)) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 577, in post return self.request('POST', url, data=data, json=json, kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 529, in request resp = self.send(prep, send_kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 645, in send r = adapter.send(request, kwargs) File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 450, in send timeout=timeout File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 710, in urlopen chunked=chunked, File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 1042, in _validate_conn conn.connect() File "/usr/local/lib/python2.7/dist-packages/urllib3/connection.py", line 358, in connect self.sock = conn = self._new_conn() File "/usr/local/lib/python2.7/dist-packages/urllib3/connection.py", line 175, in _new_conn (self._dns_host, self.port), self.timeout, extra_kw File "/usr/local/lib/python2.7/dist-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) File "/usr/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) KeyboardInterrupt