Invoke-IcingaCheckHyperVVMHealth reports zero active VMs under Microsoft Windows Server 2012 R2 Standard

[orbison]

Hi there,

the Invoke-IcingaCheckHyperVVMHealth cmdlet produces strange results when invoked under hypervisors running Windows Server 2012 R2 Standard:

PS C:\> Use-Icinga
PS C:\> Invoke-IcingaCheckHyperVVMHealth -Verbosity 3
[OK] Virtual Computers: 1 Ok (All must be [OK])
\_ [OK] Active VMs: 0
|
0

The main issue is the fact that there are actually dozens of active VMs on each of these hypervisors.

Another issue is the discrepancy between "Virtual Computers: 1 Ok", "Active VMs: 0", and reality (20 active VMs). But that seems to have already been reported in #39.

We have eight Hyper-V servers 2012 R2 Standard and the behaviour is the same on all of them. Hypervisors running Windows Server 2016 Standard and Windows Server 2019 Standard do not have this problem.

According to the documentation, Windows Server 2012 R2 is supported.

Below please find details about the affected machines:

.NET Framework:
Name                           Property
----                           --------
1033                           Version       : 4.8.03761
                               CBS           : 1
                               TargetVersion : 4.0.0
                               Install       : 1
                               Servicing     : 0
                               Release       : 528049

PowerShell:
Name                           Value
----                           -----
PSVersion                      5.1.14409.1018
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0, 5.0, 5.1.14409.1018}
BuildVersion                   10.0.14409.1018
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

OS:
OsName                                                  : Microsoft Windows Server 2012 R2 Standard
OsType                                                  : WINNT
OsOperatingSystemSKU                                    : StandardServerEdition
OsVersion                                               : 6.3.9600
OsBuildNumber                                           : 9600
OsHardwareAbstractionLayer                              : 6.3.9600.18969
OsArchitecture                                          : 64-bit
OsLanguage                                              : en-US
OsPrimary                                               : True
OsProductType                                           : Server
OsServerLevel                                           : FullServer

Please let me know if any further information is needed to analyze and resolve the issue.

Thanks,

Rostislav

[LordHepipud]

Sorry for the late response. What happens if you run the command locally of an administrative shell? I assume this is a permission issue, which causes the local user/Icinga Agent to not be able to fetch these information.

You will either have to use Icinga for Windows JEA Profiles or grant the user the Icinga Agent is running with permissions to access Hyper-V information.

[orbison]

@LordHepipud, thanks for your advice. We have tried running the IcingaCheckHyperVVMHealth check under an elevated shell and under local admin shell, but the results were the same, zero VMs reported while in fact there are about 20 of them.

Is it worth trying to use the JEA Profiles under these circumstances?

[LordHepipud]

You can give it a try. In general this should resolve the problem (was the solution in other environments).

Are the users you are trying to fetch the data from inside the Hyper-V Administrator group? If not, then you will not see any information. For JEA, this would not be required.

[orbison]

Yeah, both users we tried are local admins so they have Hyper-V-related permissions. We have even tried adding them to the respective user group, but to no avail. We'll try the JEA approach. WIll keep you posted.

[orbison]

Okay, so we did our best to make the JEA running. First, we had to perform a couple of updates:

1. Icinga PS FrameWork to version 1.7.1.
2. Icinga Agent to version 2.13.1.

That should have allowed us to deploy JEA. So we tried that and these are the results:

1. A local user has been created.
2. A JEA profile has been created.
3. The script produced the following errors and warnings during its run:

   [Error]: Unable to include module "icinga-powershell-hyperv" into JEA profile. The file "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-hyperv\provider\vcomputer\Get-IcingaVirtualComputerInfo.psm1" is using one or more [ScriptBlock] variables which are forbidden in JEA context.
   [Warning]: The module "icinga-powershell-plugins" is using "Add-Type" definitions for file "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-plugins\provider\disks\Get-IcingaDiskAttributes.psm1". Ensure you validate the code before trusting this publisher.
   [Warning]: The module "icinga-powershell-plugins" is using "Add-Type" definitions for file "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-plugins\provider\disks\Get-IcingaUNCPathSize.psm1". Ensure you validate the code before trusting this publisher.
   [Error]: Unable to include module "icinga-powershell-plugins" into JEA profile. The file "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-plugins\provider\enums\Icinga_ProviderEnums.psm1" is using one or more [ScriptBlock] variables which are forbidden in JEA context.

According to the docs, we tried to carry on with opening a new PS session, but it didn't work:

PS C:\> powershell.exe -ConfigurationName 'IcingaForWindows'
Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
The shell cannot be started. A failure occurred during initialization:
Cannot create or open the configuration session IcingaForWindows.
PS C:\>

I reckon we've reached the end of the road. What do you suggest now, @LordHepipud?

[LordHepipud]

To include the Hyper-V plugins, you might require to add the -IncludeScriptBlocks argument. Then the error is mitigated and the plugins are included.

For JEA itself: Inside the root folder of the Icinga PowerShell Framework, there is a IcingaForWindows.pssc file. You will require to add the user you are currently running the PowerShell with inside the RoleDefinitions sections, similar to the one Icinga for Windows uses. Example:

RoleDefinitions       = @{
    'icinga' = @{
        'RoleCapabilities' = 'IcingaForWindows'
    }
    'your user name' = @{
        'RoleCapabilities' = 'IcingaForWindows'
    }
}

Then you need to re-install the JEA profile:

Register-PSSessionConfiguration -Name 'IcingaForWindows' -Path 'path to above pssc file' -Force;

Then try again to enter a PowerShell session with

powershell.exe -ConfigurationName 'IcingaForWindows'

[orbison]

Thanks for your suggestions, @LordHepipud.

We completely uninstalled the Icinga JEA Profile. And then installed JEA Profile again with -AllowScriptBlocks (note: all the steps were done by a colleague of mine who is in charge of Windows servers so I cannot tell right now why it was -AllowScriptBlocks and not -IncludeScriptBlocks as you've suggested).

The errors disappeared. We've got couple of warnings instead:

PS C:\Windows\system32> Use-Icinga
PS C:\Windows\system32> Install-IcingaSecurity -AllowScriptBlocks
[Notice]: Installing user "icinga"
[Notice]: User was successfully created.
[Notice]: Service User ".\icinga" for service "icinga2" successfully updated
[Notice]: Service User ".\icinga" for service "icingapowershell" successfully updated
[Passed]: Directory "C:\ProgramData\icinga2\etc" is accessible and writable by the Icinga Service User "icinga"
[Passed]: Directory "C:\ProgramData\icinga2\var" is accessible and writable by the Icinga Service User "icinga"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\cache" is accessible and writable by the Icinga Service User "icinga"
[Passed]: Directory "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\config" is accessible and writable by the Icinga Service User "icinga"
[Notice]: Restarting service "icinga2"
[Notice]: Stopping service "icingapowershell"
[Notice]: Restarting service "icingapowershell"
[Notice]: User "icinga" including permissions was successfully installed on this host
[Notice]: Writing Icinga for Windows environment information as JEA profile
[Warning]: Module "icinga-powershell-hyperv" is containing [ScriptBlock] like content inside file "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-hyperv\provider\vcomputer\Get-IcingaVirtualComputerInfo.psm1". Please validate the file before running it inside JEA context.
[Warning]: The module "icinga-powershell-plugins" is using "Add-Type" definitions for file "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-plugins\provider\disks\Get-IcingaDiskAttributes.psm1". Ensure you validate the code before trusting this publisher.
[Warning]: Module "icinga-powershell-plugins" is containing [ScriptBlock] like content inside file "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-plugins\provider\disks\Get-IcingaPartitionSpace.psm1". Please validate the file before running it inside JEA context.
[Warning]: The module "icinga-powershell-plugins" is using "Add-Type" definitions for file "C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-plugins\provider\disks\Get-IcingaUNCPathSize.psm1". Ensure you validate the code before trusting this publisher.
[Notice]: Registering Icinga for Windows JEA profile
[Notice]: JEA Profile "IcingaForWindows" was successfully installed
[Notice]: Stopping service "icingapowershell"
[Notice]: Restarting service "icingapowershell"
PS C:\Windows\system32>

After adding RoleDefinitions into the IcingaForWindows.pssc file and running Icinga JEA profile re-registration we can now successfully invoke PowerShell with the Icinga JEA Profile:

PS C:\> Register-PSSessionConfiguration -Name 'IcingaForWindows' -Path 'C:\Program Files\WindowsPowerShell\Modules\icinga-powershell-framework\IcingaForWindows.pssc' -Force
   WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Plugin
Type            Keys                                Name
----            ----                                ----
Container       {Name=IcingaForWindows}             IcingaForWindows
PS C:\>
PS C:\> powershell.exe -ConfigurationName 'IcingaForWindows'
Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
PS>

Unfortunately the original problem remains unsolved:

PS>Use-Icinga
PS>Get-Service vmms
Status   Name               DisplayName
------   ----               -----------
Running  vmms               Hyper-V Virtual Machine Management
PS>Invoke-IcingaCheckHyperVVMHealth -Verbosity 3
[OK] Virtual Computers: 1 Ok (All must be [OK])
\_ [OK] Active VMs: 0
|
0
PS>
PS>

[orbison]

Furthermore — on this particular machine — agent-based checks started misbehaving. Performance data are no longer available for them and the pipe symbol gets displayed instead:

$ icingacli monitoring list services --host=VRTOV.example.com --verbose

   UP    VRTOV.example.com: PING OK - Packet loss = 0%, RTA = 8.27 ms
   OK    ├─ NSClient++ Version (For 1d 8h)
         │  0.5.2.39 2018-02-04|
         │  
   OK    ├─ Windows Details (Since Nov 19)
         │  Microsoft Windows Server 2012 R2 Standard, <NULL>, VRTOV, 64-bit, 3, 6.3.9600|
         │  
   OK    ├─ Windows Version (Since Nov 23)
         │  OK: Windows Server 2012 R2 (6.3.9600)
         │  
   OK    ├─ Windows Build Number (Since Nov 19)
         │  9600|
         │  
   OK    ├─ Windows Disk Space (Since Nov 25)
         │  |
         │  
   OK    ├─ Windows CPU Load (Since Nov 25)
         │  |
         │  
   OK    ├─ TEST-VMs (Since Nov 25)
         │  |
         │  
   OK    ├─ Windows Memory Usage (Since Nov 25)
         │  |
         │  
   OK    ├─ Windows Uptime (Since Nov 25)
         │  UPTIME OK 1631030s 
         │  
   OK    ├─ Hyper-V Services (Since Nov 25)
         │  |
         │  
   OK    ├─ Number of CPU Cores (Since Nov 19)
         │  4|
         │  
   OK    ├─ Splunk Forwarder (Since Nov 25)
         │  |
         │  
   CRIT  └─ Hardware Warranty (Since Sep 20)
            HW warranty expired on 2021-02-10 (305 days ago)

Cf another host without JEA Profile:

$ icingacli monitoring list services --host=VRTBM.example.com --verbose

   UP    VRTBM.example.com: PING OK - Packet loss = 0%, RTA = 11.82 ms
   OK    ├─ NSClient++ Version (Since Nov 23)
         │  0.5.2.39 2018-02-04
         │  
   OK    ├─ Windows Details (Since Sep 22)
         │  Microsoft Windows Server 2012 R2 Standard, <NULL>, VRTBM, 64-bit, 3, 6.3.9600
         │  
   OK    ├─ Windows Version (Since Sep 22)
         │  OK: Windows Server 2012 R2 (6.3.9600)
         │  
   OK    ├─ Windows Build Number (Since Nov 23)
         │  9600
         │  
   OK    ├─ Windows Disk Space (Since Nov 23)
         │  [OK] Used Partition Space: 3 Ok
         │  
   OK    ├─ Windows CPU Load (Since Nov 23)
         │  [OK] CPU Load
         │  
   OK    ├─ TEST-VMs (Since Nov 23)
         │  [OK] Virtual Computers: 1 Ok (All must be [OK])
         │  
   OK    ├─ Windows Memory Usage (Since Nov 23)
         │  [OK] Memory Usage
         │  
   OK    ├─ Windows Uptime (For 34m 20s)
         │  UPTIME OK 1625450s 
         │  
   OK    ├─ Hyper-V Services (Since Nov 23)
         │  [OK] Services: 1 Ok (All must be [OK])
         │  
   OK    ├─ Number of CPU Cores (Since Oct 21)
         │  4
         │  
   OK    ├─ Splunk Forwarder (Since Oct 21)
         │  [OK] Services: 1 Ok
         │  
   CRIT  └─ Hardware Warranty (Since Sep 20)
            HW warranty expired on 2021-02-10 (305 days ago)

The following table summarises Service checks with their check commands and provides information on displayed results:

                                                                Perfdata Perfdata Vertical bar
Service                 Command                                 expected shown    displayed
----------------------- --------------------------------------- -------- -------- ------------
NSClient++ Version      nscp.exe                                no       no       yes
Windows Details         nscp.exe                                no       no       yes
Windows Version         nscp.exe                                yes      yes      no
Windows Build Number    nscp.exe                                no       no       yes
Windows Disk Space      Invoke-IcingaCheckUsedPartitionSpace    yes      no       yes
Windows CPU Load        Invoke-IcingaCheckCPU                   yes      no       yes
TEST-VMs                Invoke-IcingaCheckHyperVVMHealth        no       no       yes
Windows Memory Usage    Invoke-IcingaCheckMemory                yes      no       yes
Windows Uptime          check_uptime.exe                        yes      yes      no
Hyper-V Services        Invoke-IcingaCheckService               yes      no       yes
Number of CPU Cores     nscp.exe                                no       no       yes
Splunk Forwarder        Invoke-IcingaCheckService               yes      no       yes
Hardware Warranty       <passive check>                         yes      yes      no

It's apparent that:

• Icinga Agent check commands (Invoke-Icinga*) stopped producing performance data. (Or perhaps Icinga stopped detecting/parsing/interpreting them and displays the vertical bar instead.)
• Performance data are processed and displayed correctly if they come from external programs (nscp.exe, check_uptime.exe) or from passive checks via REST API.
• When performance data is not available, the vertical bar appears in the plugin output.

[LordHepipud]

This is a very weird behavior. I honestly have no idea where to start in this case.

If you run these commands directly, what is the result?

# Get all information about the hyperv virtual Computers
Get-IcingaWindowsInformation -ClassName Msvm_SummaryInformation -Namespace 'Root\Virtualization\v2';

# Gather all information regarding virtual computers memory allocation
Get-IcingaWindowsInformation -ClassName Msvm_MemorySettingData -Namespace 'Root\Virtualization\v2';

# Get infos about virtual computers hard disk drive / VHDs
Get-IcingaWindowsInformation -ClassName Msvm_LogicalDisk -Namespace 'Root\Virtualization\v2';

# Get the Partition where all virtual Computers are Stored
Get-IcingaWindowsInformation -ClassName Msvm_StorageAllocationSettingData -Namespace 'Root\Virtualization\v2';

[orbison]

Yeah, I do appreciate your honesty, @LordHepipud :+1: This problem is really annoying.

Attached please find the output of the four commands you suggested just before Christmas :smiley:

VRTOV-ICINGA-CMDLETS.txt

Hope this helps...

[LordHepipud]

Thank you for the information. Can you get more into detail, on how this hypervisor is configured?

The problem is that the command

Get-IcingaWindowsInformation -ClassName Msvm_SummaryInformation -Namespace 'Root\Virtualization\v2';

is not returning any data. This is actually required, because this should contain all virtual machines. The other information are just additional details. That is why the plugin is not reporting any virtual machines.

[orbison]

Please note that we have been observing the same behaviour on all hypervisors running Windows Server 2012 R2 Standard. No other hypervisor running different version of Windows Server exhibits this behaviour.

All hypervisors are configured more or less the same. Please check out my original report for more details on the hypervisor configuration. What other particular pieces of information would be helpful?

[LordHepipud]

I was just wondering if the Hyper-V is run on a certain type of cluster, which might result in an inactive/passive side to now provide any virtual machine information. I will have to setup a Windows 2012 R2 test environment with Hyper-V, to check if this is a OS and version problem and how it could be resolved.

[orbison]

An update from the hypervisors' administrator:

• There is no cluster, these are all standalone servers.
• We can provide you with an access to a testing hypervisor via TeamViewer. If you'd like to check out and troubleshoot the real environment, I'd be happy to arrange the access for you.

[dusanbruna]

Hello, I cooperate with @orbison on solving this problem. According to https://github.com/Icinga/icinga-powershell-hyperv/issues/44#issuecomment-1074060641 I tried to run these PS commands on our affected system (logged in as a local built-in administrator, elevated Powershell):

PS C:\Windows\system32> Use-Icinga
PS C:\Windows\system32> Get-IcingaWindowsInformation -ClassName Msvm_SummaryInformation -Namespace 'Root\Virtualization\v2'
PS C:\Windows\system32>
PS C:\Windows\system32> Get-IcingaWindowsInformation -ClassName Msvm_VirtualSystemManagementService -Namespace 'Root\Virtualization\v2'

Caption                  : Hyper-V Virtual System Management Service
Description              : Service for creating, manipulating, and managing virtual systems
ElementName              : Hyper-V Virtual System Management Service
InstanceID               :
CommunicationStatus      :
DetailedStatus           :
HealthState              : 5
InstallDate              : 7. 3. 2016 12:40:22
Name                     : vmms
OperatingStatus          :
OperationalStatus        : {2}
PrimaryStatus            :
Status                   : OK
StatusDescriptions       : {The service is running normally}
AvailableRequestedStates :
EnabledDefault           : 2
EnabledState             : 2
OtherEnabledState        :
RequestedState           : 12
TimeOfLastStateChange    : 16. 3. 2022 20:11:11
TransitioningToState     :
CreationClassName        : Msvm_VirtualSystemManagementService
PrimaryOwnerContact      :
PrimaryOwnerName         :
Started                  : True
StartMode                :
SystemCreationClassName  : Msvm_ComputerSystem
SystemName               : VRTBM
PSComputerName           :

PS C:\Windows\system32>

The class Msvm_SummaryInformation (the description is Used in the GetSummaryInformation method in the Msvm_VirtualSystemManagementService class to quickly retrieve common information related to a virtual system or snapshot.) returns no information.

The class Msvm_VirtualSystemManagementService (the description is Represents the virtualization service present on a single host system. Msvm_VirtualSystemManagementService is used to control the definition, modification, and deletion of virtual systems. It also has methods for performing operations on virtual systems, such as cloning, snapshotting, and the importing or exporting of virtual systems. To retrieve per-VM information, use Msvm_ComputerSystem and Msvm_PlannedComputerSystem.) returns correctly.

From my point of view is this the permission problem in the GetSummaryInformation method for the Msvm_VirtualSystemManagementService class (according to its description).

@LordHepipud could you help how to chceck (or how to set) the proper permission for that?

Thanks, Dusan

[LordHepipud]

Thank you for the input. I was already testing on a Windows 2012 R2 and I can confirm, that Msvm_SummaryInformation is not returning any information.

I already tried to set different permissions, followed by a reboot and yet no result on the output. I even updated to a newer Management Framework Version (5.1) to run the plugin in JEA context, without success either.

Either this class is not properly supported by Windows 2012 R2 or something minor is missing here.

[orbison]

It's good to see that you were able to confirm and replicate our issue. The ticket is sill labelled needsfeedback. So is there anything else we can do to help you to find a solution to this long-lasting problem?

[LordHepipud]

Not at the moment. I can't figure out a good alternative to fetch the required information from Windows 2012 R2. In general as far as I can tell, WMI should work but for some reason isn't. Anyone got an idea?