search

alanrenouf / vCheck-vSphere

vCheck Daily Report for vSphere
MIT License
733 stars 325 forks source link

vCheck hangs after vCenter Uprade to 7.0U3c #743

Open VM-Master opened 2 years ago

VM-Master commented 2 years ago

Hi guys,

we updated our vCenter last weekend from 7.0.2 18455184 to latest version 7.0.3 19234570. Now I could see that plugin "33 VM CPU Usage" will not finished anymore. If I disabled these plugin all other enabled plugins runs like before. Our vCenter has approx. 500 VMs. It's a vCenter for our VMware Horizon VDIs.

Any ideas why this check doesn't work anymore or maybe need a lot of more time?

roydebets commented 2 years ago

Hi,

We experience the same thing, but not with al vCenters on 7.0U3c. It started with one vCenter which did not complete the vCheck, but today a 2nd vCenter does not complete the vCheck as well.

Also, it seems that it's not always the same plugin which takes too much time to complete. This seems random and even multiple plugins take long completion time.

If anyone has a solution, please share!

roydebets commented 2 years ago

I found that for a specific vCenter the vCheck used to complete in 5 minutes and now takes 67 minutes. Also, it lists the plugins that take longer than 60 seconds to complete.

09 Created or cloned VMs.ps1 1285.52 96 VMWare Replication Check.ps1 1189.46 77 Lost access to volume.ps1 655.85 01 General Information.ps1 629.19

Just collecting general info already takes more than 10 minutes. Our vChecks do not complete since we run the via scheduled task and it is set to abort of the task runs for more than 1 hour. I will extend that time now, but still strange that it takes more than 10x longer to complete.

VM-Master commented 2 years ago

I don't have plugin "96" and "09" enabled in my environment. Plugin 01 took on vCenter 7 U1 100seconds, now on 7U3c it takes 210 seconds.

Plugin "77" seems to have an issue in my environment... ++++++++++++++++++++++++++++++++++++ [03:39:51] ..start calculating Lost Access to Volume by Robert van den Nieuwendijk, Jonathan Medd v1.1 [32 of 89] PS>TerminatingError(Get-VIEvent): "Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again." Get-VIEvent : Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again. At D:\Scripts\vCheckVDI\Plugins\30 Host\77 Lost Access to Volume.ps1:9 char:83

[03:39:51] ..finished calculating Lost Access to Volume by Robert van den Nieuwendijk, Jonathan Medd v1.1 [32 of 89] ++++++++++++++++++++++++++++++++++++

roydebets commented 2 years ago

Digging into it deeper I see various plugins having error messages about "Get-VIEvent". When checking the events on the vCenter manually, there are just no events to display. We have now raised a ticket at VMware for this.

We have upgraded various vCenters to 7u3c, but only 3 of them have issues.

Van: VM-Master @.> Beantwoorden - Aan: alanrenouf/vCheck-vSphere @.> Datum: woensdag 16 maart 2022 om 12:04 Aan: alanrenouf/vCheck-vSphere @.> CC: Roy Debets @.>, Comment @.***> Onderwerp: Re: [alanrenouf/vCheck-vSphere] vCheck hangs after vCenter Uprade to 7.0U3c (Issue #743)

I don't have plugin "96" and "09" enabled in my environment. Plugin 01 took on vCenter 7 U1 100seconds, now on 7U3c it takes 210 seconds.

Plugin "77" seems to have an issue in my environment... ++++++++++++++++++++++++++++++++++++ [03:39:51] ..start calculating Lost Access to Volume by Robert van den Nieuwendijk, Jonathan Medd v1.1 [32 of 89] PS>TerminatingError(Get-VIEvent): "Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again." Get-VIEvent : Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again. At D:\Scripts\vCheckVDI\Plugins\30 Host\77 Lost Access to Volume.ps1:9 char:83

· ~~~~~ ·

 *   CategoryInfo : InvalidData: (:) [Get-VIEvent], ParameterBindingValidationException
 *   FullyQualifiedErrorId : ParameterArgumentValidationError,VMware.VimAutomation.ViCore.Cmdlets.Commands.GetEvent

Get-VIEvent : Cannot validate argument on parameter 'MaxSamples'. The argument is null, empty, or an element of the argument collection contains a null value. Supply a collection that does not contain any null values and then try the command again. At D:\Scripts\vCheckVDI\Plugins\30 Host\77 Lost Access to Volume.ps1:9 char:83

· ~~~~~ ·

 *   CategoryInfo : InvalidData: (:) [Get-VIEvent], ParameterBinding

ValidationException

[03:39:51] ..finished calculating Lost Access to Volume by Robert van den Nieuwendijk, Jonathan Medd v1.1 [32 of 89] ++++++++++++++++++++++++++++++++++++

— Reply to this email directly, view it on GitHubhttps://github.com/alanrenouf/vCheck-vSphere/issues/743#issuecomment-1069002475, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUXNEXXLLTDBZ6PDVW2RFGLVAG55DANCNFSM5QCREXYA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub. You are receiving this because you commented.Message ID: @.***>

TheGabeMan commented 2 years ago

Same here. Measure-Command{ Get-VIEvent -MaxSamples 100 -Start (get-date).AddDays(-1) } Eventually I then get an error: Get-VIEvent : 23-3-2022 09:59:10 Get-VIEvent The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. Always after 10 minutes.

But we're also seeing impact on backups wanting to create snapshots and time out waiting. Other processes also have issues, like gathering licenses via Cloud Usage Meter.

TheGabeMan commented 2 years ago

Solved by performing two actions:

VM-Master commented 2 years ago

Hi, yes, the DB optimizations seems to be helpful --> https://kb.vmware.com/s/article/87837?lang=en_US It's officially recommended for everyone who upgraded the VCSA from 7.0U2 to U3.

TheGabeMan commented 2 years ago

Case can probably be closed now.