Unable to configure vCenter

linkztream commented 4 years ago

Configuring integration through ui to connect to vsphere 6.7.0.40000 does not work. Adding the integration and selecting the host directly works fine. Log details says:

Config entry for esxi_stats not ready yet. Retrying in 10 seconds.

config_entries.py (WARNING) - Config entry for esxi_stats not ready yet. Retrying in 10 seconds. unsupported operand type(s) for /: 'NoneType' and 'int' custom_components/esxi_stats/__init__.py **To Reproduce** Steps to reproduce the behavior: 1. Run Hass.io 2. Install component through HACS 3. Configure Integration through UI 4. See error in logs and no sensors showing up **Expected behavior** Sensors for my 2 vmware hosts showing up in HA **ESXi environment:** - vSphere 6.7.0.40000 deployed from appliance - 2 hosts running VMware ESXi, 6.5.0, 5969303 - vCenter standard license - Self-Signed certificate **Additional context** Tested running the integration both as dedicated hassio user and default vsphere admin user. Hass.io running dockerized on Ubuntu 18.04 VM, vSphere is appliance downloaded from vmware.com

wxt9861 commented 4 years ago

Can you enabled debug logging and provide esxi debug logs during hass startup?

logger:
  logs:
    custom_components.esxi_stats: debug

linkztream commented 4 years ago

Sure! Here: home-assistant.log

linkztream commented 4 years ago

Looking at the logs it occured to me that two pieces of fact are missing:

The login for vCenter is not the same as the login on the hosts.
10.0.0.6 is in standby (automatic) and it never goes on to check 10.0.0.7

I'll wake 10.0.0.6 to see if it works better then.

linkztream commented 4 years ago

Yep, that did it. Now that both hosts are awake and responding, it works great and sensors for both hosts, datastores, vcenter etc are available, so it looks like the empty reply from a disconnected/offline host causes further execution to fail.

linkztream commented 4 years ago

Just to be clear: The whole integration dies again when the host goes back into standby.

wxt9861 commented 4 years ago

Ok, I think the way to solve this is to check if the host is online before trying to pull stats from it. I don't have vcenter to test this on. Will you be able to help me test?

linkztream commented 4 years ago

Absolutely, let me know what you need. :)

wxt9861 commented 4 years ago

I made some initial changes that should help us to a resolution. Please install this beta (you should be able to install via HACS if you enable beta releases) and give it a try. Run it with debug, so we can catch logs.

This releases adds a check of the hosts runtime state before pulling other stats. If the host is not poweredOn, we're not pulling any stats and instead filling in "n/a". If this works, you should have a sensor that only contains the host's name, it's state (i am assuming standBy) and the rest of the attributes will say "n/a" until the host is powered on.

linkztream commented 4 years ago

Sorry for the late reply, been a little hectic but now I can focus on this. :)

Tried it out, and it feels like it was real close but it won't go on to 10.0.0.7 still.

home-assistant.log

wxt9861 commented 4 years ago

Let me understand this.

Your vcenter is x.x.16.50 Your hosts are x.x.0.6 and x.x.0.7

We can see that x.x.0.6 is in standby, so we're skipping that host, however I am not seeing anything for x.x.0.7. Is that the running host, and also connected to vCenter?

linkztream commented 4 years ago

Exactly right. 👍

vCenter is hell bent on putting the first host into standby, probably because the first host has 96GB RAM and the 2nd 128GB (memory was cheaper when the 2nd was added).

vCenter is a virtual machine, and it's running on the host that stays up.

Feels abit like execution goes to x.x.0.6, figures it's down/standby and thinks "Meh!" and aborts instead of moving on to .7?

wxt9861 commented 4 years ago

Sorry for delay. Yeah, that seems to be whats happening but it should keep going. Let me make a script (by passing HA) to run against the vcenter to verify we're actually seeing 2 hosts and see if we can get info or an error. Stay tuned.

linkztream commented 4 years ago

Just a quick follow up question, are the hosts queried directly even though connection is made to vCenter?

wxt9861 commented 4 years ago

Since connected to a vcenter, the data is queried from vcenter. If you create an integration to the host directly, it'll talk to the server directly.

wxt9861 commented 4 years ago

Are you able to run a python script from your HA install? If so, please run this against your vcenter so we can figure out what information it is pulling.

This will connect to vcenter and create a hosts object view (just like the component does) and return that information back to you. What we're looking for here is how many hosts it finds (should be 2) and does it actually pull information from both hosts or what?

The file is in .txt format, but rename it to .py format. I can't attach .py files here You'll need to pass your vcenter login to the script for it to connect, so the command should look like this: python3 hostCheck.py -s vcenter_ip -u username -p password

If you run it successfully, post the output here (remove/replace any confidential information)

hostCheck.txt

linkztream commented 4 years ago

Thanks! I'm running all docker hassio, but I'll improvie, adapt and overcome. I'll get back to you with a log in a couple of hours when I get home.

linkztream commented 4 years ago

Logged in - session 5266354c-ca4e-a384-6235-c16a98fa1a18 Number of hosts found: 2 (ManagedObject) [ 'vim.HostSystem:host-263', 'vim.HostSystem:host-182' ] 10.0.0.6 vmhost: 10.0.0.6 state is standBy Unable to return stats for 10.0.0.6 Logged out - session 5266354c-ca4e-a384-6235-c16a98fa1a18 Traceback (most recent call last): File "hostCheck.py", line 157, in main() File "hostCheck.py", line 146, in main check = get_host_info(esxi_host) File "hostCheck.py", line 111, in get_host_info "maintenance_mode": host_mm_mode, UnboundLocalError: local variable 'host_mm_mode' referenced before assignment

wxt9861 commented 4 years ago

Thanks. So looking at this, we're know it is seeing 2 host objects, but it is only scanning one host and then logging out. Let me figure out whats going on here.

wxt9861 commented 4 years ago

Here is another script. This is going to help us determine if the issue is in the loop statement. hostCheck.txt

linkztream commented 4 years ago

Logged in - session 5217510b-a523-8be3-04f8-7ae2c02ab40c
Number of hosts found: 2
(ManagedObject) [
   'vim.HostSystem:host-263',
   'vim.HostSystem:host-182'
]
Checking using array #0
vmhost: 10.0.0.6 state is standBy
Unable to return stats for 10.0.0.6
{'name': '10.0.0.6', 'state': 'standBy', 'version': 'n/a', 'uptime_hours': 'n/a', 'cputotal_ghz': 'n/a', 'cpuusage_ghz': 'n/a', 'memtotal_gb': 'n/a', 'memusage_gb': 'n/a', 'vms': 'n/a'}
Checking using array #1
vmhost: 10.0.0.7 state is poweredOn
{'name': '10.0.0.7', 'state': 'poweredOn', 'version': '6.5.0', 'uptime_hours': 5831.9, 'cputotal_ghz': 31.9, 'cpuusage_ghz': 4.6, 'memtotal_gb': 127.99, 'memusage_gb': 92.63, 'vms': 16}
-----------------------
Checking using for loop
10.0.0.6
vmhost: 10.0.0.6 state is standBy
Unable to return stats for 10.0.0.6
{'name': '10.0.0.6', 'state': 'standBy', 'version': 'n/a', 'uptime_hours': 'n/a', 'cputotal_ghz': 'n/a', 'cpuusage_ghz': 'n/a', 'memtotal_gb': 'n/a', 'memusage_gb': 'n/a', 'vms': 'n/a'}
This FOR statement looped: 1
10.0.0.7
vmhost: 10.0.0.7 state is poweredOn
{'name': '10.0.0.7', 'state': 'poweredOn', 'version': '6.5.0', 'uptime_hours': 5831.9, 'cputotal_ghz': 31.9, 'cpuusage_ghz': 4.6, 'memtotal_gb': 127.99, 'memusage_gb': 92.63, 'vms': 16}
This FOR statement looped: 2
Logged out - session  5217510b-a523-8be3-04f8-7ae2c02ab40c

linkztream commented 4 years ago

This last log looks better to me than the one before. Now the "Logged out" doesn't happen until after processing 10.0.0.7, or am I wrong?

wxt9861 commented 4 years ago

Very odd - this scanned both servers, like it should. However, the only thing I added was code to scan manually in addition to a loop. Do you still have the original script that you can run? Maybe the maintenance mode is throwing it off. Can you run both one after another? I think we're getting to the root of the problem.

linkztream commented 4 years ago

Original script:

Logged in - session 52555aec-6d1f-03c8-ae0d-65f10f00f418
Number of hosts found: 2
(ManagedObject) [
   'vim.HostSystem:host-263',
   'vim.HostSystem:host-182'
]
10.0.0.6
vmhost: 10.0.0.6 state is standBy
Unable to return stats for 10.0.0.6
Logged out - session  52555aec-6d1f-03c8-ae0d-65f10f00f418
Traceback (most recent call last):
  File "hostCheck_org.py", line 157, in <module>
    main()
  File "hostCheck_org.py", line 146, in main
    check = get_host_info(esxi_host)
  File "hostCheck_org.py", line 111, in get_host_info
    "maintenance_mode": host_mm_mode,
UnboundLocalError: local variable 'host_mm_mode' referenced before assignment

New script:

Logged in - session 5258c097-8e85-e965-c2d7-fba5db37dbf3
Number of hosts found: 2
(ManagedObject) [
   'vim.HostSystem:host-263',
   'vim.HostSystem:host-182'
]
Checking using array #0
vmhost: 10.0.0.6 state is standBy
Unable to return stats for 10.0.0.6
{'name': '10.0.0.6', 'state': 'standBy', 'version': 'n/a', 'uptime_hours': 'n/a', 'cputotal_ghz': 'n/a', 'cpuusage_ghz': 'n/a', 'memtotal_gb': 'n/a', 'memusage_gb': 'n/a', 'vms': 'n/a'}
Checking using array #1
vmhost: 10.0.0.7 state is poweredOn
{'name': '10.0.0.7', 'state': 'poweredOn', 'version': '6.5.0', 'uptime_hours': 5850.4, 'cputotal_ghz': 31.9, 'cpuusage_ghz': 6.2, 'memtotal_gb': 127.99, 'memusage_gb': 92.63, 'vms': 16}
-----------------------
Checking using for loop
10.0.0.6
vmhost: 10.0.0.6 state is standBy
Unable to return stats for 10.0.0.6
{'name': '10.0.0.6', 'state': 'standBy', 'version': 'n/a', 'uptime_hours': 'n/a', 'cputotal_ghz': 'n/a', 'cpuusage_ghz': 'n/a', 'memtotal_gb': 'n/a', 'memusage_gb': 'n/a', 'vms': 'n/a'}
This FOR statement looped: 1
10.0.0.7
vmhost: 10.0.0.7 state is poweredOn
{'name': '10.0.0.7', 'state': 'poweredOn', 'version': '6.5.0', 'uptime_hours': 5850.4, 'cputotal_ghz': 31.9, 'cpuusage_ghz': 6.2, 'memtotal_gb': 127.99, 'memusage_gb': 92.63, 'vms': 16}
This FOR statement looped: 2
Logged out - session  5258c097-8e85-e965-c2d7-fba5db37dbf3

Looks like you're right. :)

wxt9861 commented 4 years ago

Can you try this beta release? It addresses maintenance mode issue, and hopefully fixes the problem https://github.com/wxt9861/esxi_stats/releases/tag/0.5.2b2

linkztream commented 4 years ago

Sorry, no real difference. :(

I'm not a programmer, so I don't know what the pyvmomi code says, but the host is not in maintenance mode, it's in standby (quite different actually). Could that be a clue?

Maintenance mode = powered on and responding to most commands/queries, but unavailable for VM operations. Used for patching etc. This is the mode you should pass through before rebooting or powering off a host in a cluster to trigger vMotion etc to move all active VM's to other hosts in the cluster.
Standby = Powered off/hibernating (vCenter controlled based on cluster load), but available to be brought back online in case of load increase. Does not pass through maintenance mode, and doesn't present host as disconnected. Of course, it still migrates all VM's over to an active host before entering standby and back again on wake up. Depending on your hardware, the host will be completely "dead" as the power on/off is controlled through the management interface (HP iLO or Dell iDrac just to name two of them) and this is the reason why my first host is not even responding to ping at 10.0.0.6

2020-01-25 10:00:03 DEBUG (MainThread) [custom_components.esxi_stats] Setting up host 10.0.16.50 2020-01-25 10:00:03 DEBUG (MainThread) [custom_components.esxi_stats.esxi] Logged in - session 5227997c-4934-0ee5-85a7-29ef3a8e00a9 2020-01-25 10:00:03 DEBUG (MainThread) [custom_components.esxi_stats] Product Line: vpx 2020-01-25 10:00:03 DEBUG (MainThread) [custom_components.esxi_stats.esxi] Checking license type 2020-01-25 10:00:03 DEBUG (MainThread) [custom_components.esxi_stats.esxi] Found VMware VirtualCenter Server license 2020-01-25 10:00:04 DEBUG (MainThread) [custom_components.esxi_stats.esxi] Logged in - session 520e7053-f00c-df28-631c-f01d9d4d63bc 2020-01-25 10:00:04 DEBUG (MainThread) [custom_components.esxi_stats] Getting stats for vmhost: 10.0.0.6 2020-01-25 10:00:04 DEBUG (MainThread) [custom_components.esxi_stats.esxi] vmhost: 10.0.0.6 state is standBy 2020-01-25 10:00:04 DEBUG (MainThread) [custom_components.esxi_stats.esxi] Unable to return stats for 10.0.0.6 2020-01-25 10:00:04 DEBUG (MainThread) [custom_components.esxi_stats.esxi] Logged out - session 520e7053-f00c-df28-631c-f01d9d4d63bc 2020-01-25 10:00:04 ERROR (MainThread) [custom_components.esxi_stats] local variable 'host_mm_mode' referenced before assignment 2020-01-25 10:00:04 DEBUG (MainThread) [custom_components.esxi_stats.esxi] Logged out - session 5227997c-4934-0ee5-85a7-29ef3a8e00a9 2020-01-25 10:00:04 WARNING (MainThread) [homeassistant.config_entries] Config entry for esxi_stats not ready yet. Retrying in 5 seconds.

wxt9861 / esxi_stats

Unable to configure vCenter #20