search

atc0005 / check-vmware

Go-based tooling to monitor VMware environments; NOT affiliated with or endorsed by VMware, Inc.
MIT License
17 stars 3 forks source link

Incorporate `vmware/govmomi/vcsim` in tests #585

Open atc0005 opened 2 years ago

atc0005 commented 2 years ago

Some development efforts are proving challenging as I don't have a lab environment that I can test against. In particular, I find myself needing to set Custom Attributes in vCenter and I don't have a test environment where I can safely do so.

Resources

vcsim URLs

atc0005 commented 2 years ago

Short version for exporting a vSphere inventory:

cd /tmp
wget https://github.com/vmware/govmomi/releases/download/v0.27.2/govc_Linux_x86_64.tar.gz
tar zfx govc_Linux_x86_64.tar.gz 
mv govc $HOME/go/bin/
wget https://github.com/vmware/govmomi/releases/download/v0.27.2/vcsim_Linux_x86_64.tar.gz
tar zxf vcsim_Linux_x86_64.tar.gz 
mv vcsim $HOME/go/bin

export GOVC_URL=https://vc1.example.com/sdk
export GOVC_USERNAME=vc1-read-only-service-account
export GOVC_PASSWORD=placeholder
export GOVC_INSECURE=1

$HOME/go/bin/govc object.save

I'm happy to see that a 0006-CustomFieldsManager-CustomFieldsManager.xml was created containing Custom Attributes defined within the inventory of the vCenter instance I exported.

atc0005 commented 2 years ago 
export GOVC_URL=https://vc1.example.com/sdk
export GOVC_USERNAME=vc1-read-only-service-account
export GOVC_PASSWORD=placeholder
export GOVC_INSECURE=1

With those environment variables defined, I'm able to list Custom Attributes of the live vCenter instance via govc fields.ls.

atc0005 commented 2 years ago

Short version for exporting a vSphere inventory:

cd /tmp
wget https://github.com/vmware/govmomi/releases/download/v0.27.2/govc_Linux_x86_64.tar.gz
tar zfx govc_Linux_x86_64.tar.gz 
mv govc $HOME/go/bin/
wget https://github.com/vmware/govmomi/releases/download/v0.27.2/vcsim_Linux_x86_64.tar.gz
tar zxf vcsim_Linux_x86_64.tar.gz 
mv vcsim $HOME/go/bin

export GOVC_URL=https://vc1.example.com/sdk
export GOVC_USERNAME=vc1-read-only-service-account
export GOVC_PASSWORD=placeholder
export GOVC_INSECURE=1

$HOME/go/bin/govc object.save

The next step is to start vcsim like so to load the inventory:

$ vcsim -load vcsim-vc1.example.com
export GOVC_URL=https://user:pass@127.0.0.1:8989/sdk GOVC_SIM_PID=4073

and from another console tab I ran:

$ export GOVC_URL=https://user:pass@127.0.0.1:8989/sdk GOVC_SIM_PID=4073
$ govc fields.ls
govc: Post "https://127.0.0.1:8989/sdk": x509: certificate signed by unknown authority
$ export GOVC_INSECURE=1
$ govc fields.ls
...

The expected Custom Attributes were listed.

atc0005 commented 2 years ago

Using the v0.28.1 copy of check_vmware_tools I was able to connect to the vcsim instance and get expected results like so (which matched the real environment):

$ ./check_vmware_tools-v0.28.1-0-g2cb33db-linux-amd64 --username bob --password apples --server 127.0.0.1 --port 8989 --trust-cert
{"level":"error","version":"check-vmware v0.28.1-0-g2cb33db (https://github.com/atc0005/check-vmware)","logging_level":"info","plugin_type":"vmware-tools","plugin_timeout":"10s","username":"bob","user_domain":"","trust_cert":true,"server":"127.0.0.1","port":8989,"included_resource_pools":"","excluded_resource_pools":"","num_excluded_resource_pools":0,"num_included_resource_pools":0,"ignored_vms":"","eval_powered_off":false,"vms_total":81,"vms_filtered":27,"vms_excluded_by_name":0,"vms_excluded_by_power_state":54,"vms_with_tools_issues":6,"vms_without_tools_issues":21,"resource_pools_evaluated":1,"time":"2022-01-04T06:54:34-06:00","caller":"github.com/atc0005/check-vmware/cmd/check_vmware_tools/main.go:310","message":"issues with VMware Tools found"}
CRITICAL: 6 VMs with VMware Tools issues detected (evaluated 27 VMs, 1 Resource Pools) 

**ERRORS** 

* 6 of 27 VMs with VMware Tools issues 

**THRESHOLDS** 

* CRITICAL: VMware Tools not running, not installed, unsupported or blacklisted version. 
* WARNING: Outdated VMware Tools installation. 

**DETAILED INFO** 

* 01) vm1 (poweredOn, guestToolsNotInstalled) 
* 02) vm2 (poweredOn, guestToolsSupportedOld) 
* 03) vm3 (poweredOn, guestToolsSupportedOld) 
* 04) vm4 (poweredOn, guestToolsSupportedOld) 
* 05) vm5 (poweredOn, guestToolsSupportedOld) 
* 06) vm6 (poweredOn, guestToolsSupportedOld) 

--- 

* vSphere environment: https://127.0.0.1:8989/sdk 
* Plugin User Agent: check-vmware/v0.28.1-0-g2cb33db 
* VMs (evaluated: 27, total: 81) 
* Powered off VMs evaluated: false 
* Specified VMs to exclude (0): [] 
* Specified Resource Pools to explicitly include (0): [] 
* Specified Resource Pools to explicitly exclude (0): [] 
* Resource Pools evaluated (1): [Resources] 

 | 'vms'=81;;;; 'vms_excluded_by_name'=0;;;; 'vms_excluded_by_power_state'=54;;;; 'vms_with_tools_issues'=6;;;; 'vms_without_tools_issues'=21;;;; 'resource_pools_excluded'=0;;;; 'resource_pools_included'=0;;;; 'resource_pools_evaluated'=1;;;; 'time'=190ms;;;;

This is extremely promising.

atc0005 commented 2 years ago

Current workflow for setting up a quick test environment:

Console Tab 1

cd $HOME/Desktop
vcsim -load vcsim-vc1.example.com

Console Tab 2

# These environment variables are used by govc (natively)
export GOVC_INSECURE=1
export GOVC_URL=https://user:pass@127.0.0.1:8989/sdk

# These environment variables are used by PowerCLI commands (explicitly)
export VCSIM_SERVER=127.0.0.1
export VCSIM_PORT=8989
export VCSIM_USER=bob
export VCSIM_PASS=apples

# Use govc CLI tool to list current Custom Attributes
govc fields.ls

# Launch PowerShell 7.x for further testing
pwsh
# Use PowerCLI to list VM names & their IP Addresses
PS /home/ubuntu/Desktop> $server = Connect-VIServer -server $env:VCSIM_SERVER -User $env:VCSIM_USER -Password $env:VCSIM_PASS -Port $env:VCSIM_PORT
PS /home/ubuntu/Desktop> Get-VM | Where-Object {$_.PowerState -eq 'PoweredOn'} | Select-Object -Property Name, @{N="IP";E={@($_.Guest.IPAddress)}} | Format-Table -AutoSize

# Optional listing of supported methods and types
# $About = Invoke-RestMethod -Uri "https://$($server):$($env:VCSIM_PORT)/about" -Method Get -SkipCertificateCheck
# $About.Methods
# $About.Types

Further work:

# List Custom Attributes
PS /home/ubuntu/Desktop> Get-CustomAttribute | Format-Table -AutoSize

Key Name                                                       TargetType
--- ----                                                       ----------
339 AutoDeploy.MachineIdentity                                 VMHost
6   Location                                                   VMHost
502 Location
101 Notes                                                      VMHost
201 Tier                                                       VirtualMachine
301 com.vmware.vcIntegrity.customField.scheduledTask.action
302 com.vmware.vcIntegrity.customField.scheduledTask.signature
303 com.vmware.vcIntegrity.customField.scheduledTask.target

# Create new Custom Attributes
PS /home/ubuntu/Desktop> New-CustomAttribute -Name 'Backup Status' -TargetType 'VirtualMachine'

Key   Name                 TargetType
---   ----                 ----------
0     Backup Status        VirtualMachine

PS /home/ubuntu/Desktop> New-CustomAttribute -Name 'Last Backup' -TargetType 'VirtualMachine'
Key   Name                 TargetType
---   ----                 ----------
1     Last Backup          VirtualMachine

# Set Custom Attribute values on a VM from inventory
PS /home/ubuntu/Desktop> Set-Annotation -Entity rhelcluster2 -CustomAttribute 'Backup Status' -Value 'Backup Job ID [43915] Client: [vcapps], Backup Set: [defaultBackupSet], Subclient: [VMs With Validation] Parent Backup Job ID [43901]'

AnnotatedEntity Name                 Value
--------------- ----                 -----
rhelcluster2    Backup Status        Backup Job ID [43915] Client: [vcapps], Backup Se…

PS /home/ubuntu/Desktop> Set-Annotation -Entity rhelcluster2 -CustomAttribute 'Last Backup' -Value '2021-11-09 9:07:21 PM'
AnnotatedEntity Name                 Value
--------------- ----                 -----
rhelcluster2    Last Backup          2021-11-09 9:07:21 PM

I'll need to spend more time with govc so that I can replace the PowerShell snippets with its native support (consistency in test setup).

atc0005 commented 2 years ago

Alternative to the PowerShell snippets:

govc fields.ls
govc fields.add -type VirtualMachine 'Backup Status'
govc fields.add -type VirtualMachine 'Last Backup'
govc fields.set 'Backup Status' 'Backup Job ID [43915] Client: [vcapps], Backup Set: [defaultBackupSet], Subclient: [VMs With Validation] Parent Backup Job ID [43901]' rhelcluster2
govc fields.set 'Last Backup' '2021-11-09 9:07:21 PM' rhelcluster2
govc fields.info rhelcluster2

I'm seeing duplicate Custom Attributes listed, but this may be a bug in govc. I've opened an upstream GH issue to learn if that is the case. PowerShell doesn't show the duplicate entries when listing the newly created Custom Attributes, so it appears that only the listing aspect may have a bug.