search

alanrenouf / vCheck-vSphere

vCheck Daily Report for vSphere
MIT License
735 stars 326 forks source link

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. #217

Open ajohn24 opened 10 years ago

ajohn24 commented 10 years ago

Hi, While running the vCheck script it fails while getting the hard disk info on plugin48. Seems the earlier plugin disconnects the session to the VC server. It runs fine till earlier plugins.

Get-HardDisk : 6/12/2014 9:33:00 PM Get-HardDisk Server vc.com not connected. At C:\scripts\Core\vCheck-vSphere-master\Plugins\48 Find VM Disk Format.ps1:6 char:35

Please assist

ajohn24 commented 10 years ago

This is with vCheck6

smasterson commented 10 years ago

I am seeing similar errors as well after running the latest download on three different environments. I can't tell what exactly is going on but the errors seem to go away if I disable 44 VMKernel Warnings.

Here are some of the errors I am seeing: Get-HardDisk : 6/13/2014 4:44:37 PM Get-HardDisk The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At E:\Scripts\vCheck\vcenter\Plugins\48 Find VM Disk Format.ps1:6 char:23

Get-View : 6/13/2014 4:44:42 PM Get-View The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At E:\Scripts\vCheck\vcenter\Plugins\53 Hardware status warnings-errors.ps1:6 char:20

Get-View : 6/13/2014 4:44:42 PM Get-View View with Id
'HostHealthStatusSystem-healthStatusSystem-10' was not found on the server(s).

At E:\Scripts\vCheck\vcenter\Plugins\53 Hardware status warnings-errors.ps1:6 char:20

get-view : 6/13/2014 4:44:44 PM Get-View The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At E:\Scripts\vCheck\vcenter\Plugins\63 Snapshot Removed.ps1:13 char:12

get-view : 6/13/2014 4:44:44 PM Get-View The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At E:\Scripts\vCheck\vcenter\Plugins\65 Snapshot Created.ps1:13 char:12

get-stat : 6/13/2014 4:44:44 PM Get-Stat The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At E:\Scripts\vCheck\vcenter\Plugins\68 Disk Max Total Latency.ps1:12 char:29

Get-VirtualPortGroup : 6/13/2014 4:44:47 PM Get-VirtualPortGroup The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
At E:\Scripts\vCheck\vcenter\Plugins\98 vSwitch Security.ps1:63 char:7

As mentioned previously, if I remove/disable 44 VMKernel Warnings then I get no errors at all. Oddly I can not find anything special about the plugin or why it is now causing issues.

For others that have seen issues - can you disable/remove 44 VMKernel Warnings and see if you are still having issues?

If you have a large environment, seeing errors is not super easy via the console, it's better to pipe them to a file for easy reading - this can be done like so: PS> .\vCheck.ps1 *> output.txt

smasterson commented 10 years ago

For those curious, my setup in all three environments: PowerShell v3 PowerCLI 5.5 R2 vSphere 5.5 Update 1

smasterson commented 10 years ago

I'm getting some really odd/inconsistent results on this. The issue seems to come and go for no apparent reason (that I can tell). It may create errors 4 runs in a row and then run fine the next 3 runs. Disabling 44 VMKernel Warnings does seem to help but I can't seem to figure out why that plugin would cause any issues. I ran a few tests this morning to see if the vCenter server is actually disconnecting and in my case it is apparently not (write-host $global:DefaultVIServer prior to running each plugin). The size of the environment doesn't seem to come into play - same results on >500 VMs as <20 VMs. I did try the latest release build (6.20) and did not see any issues so it appears the issue has arose since then. The latest dev build will error 99% of the time on first run. For testing I am using a fresh download, changing 3 lines in GlobalVariables (Setup $false, vCenter address, SendEmail $false) and then running.

I'm stumped...may need to call in the big dogs...

ajohn24 commented 10 years ago

Our environment looks like: Powershell v2 VMware vSphere PowerCLI 5.0 vSphere 5.1U2

Admins, any updates?

ajohn24 commented 10 years ago

BTW, the solution to disable vmkernel warning plugin does not apply since I have the plugin already disabled. And I still fail to understand how disabling the plugin would sort this issue.

ajohn24 commented 10 years ago

I tried running it again and this time it is stuck on calculating VM CPU usage. Last time also I had to kill and re-run the script while it was stuck on calculating VM CPU ready%. SO there is more than one thing which needs attention.

10:00:02 PM ..start calculating VM CPU %RDY by Alan Renouf v1.1 [35 of 84] 10:14:57 PM ..finished calculating VM CPU %RDY by Alan Renouf v1.1 [35 of 84] 10:14:57 PM ..start calculating VM CPU Usage by Alan Renouf, Sam McGeown v1.3 [36 of 84] Killed on 11:17P.M

alanrenouf commented 10 years ago

Looks like this is a .net issue, I'm wondering if there is some way to reinstall .net 3.5 or fix it?

alanrenouf commented 10 years ago

Have you tried running the script from a different machine?

ajohn24 commented 10 years ago

I'll try re-installing .net today and update. Will try on a different machine too and see if it works.

smasterson commented 10 years ago

My results are crazy inconsistent. So far with todays build, I am not able to produce any errors. Nor is my older build producing errors any longer... Is it crazy to think I just needed a reboot? I dunno

ajohn24 commented 10 years ago

For datastore information plugin, can we ignore alerts for a array of datastore. I wanted to use wildcards but it is not working $DatastoreIgnore ="local"

smasterson commented 10 years ago

@ajohn24 - the variable is regex (-notmatch) - please open a new issue for this if you are still having issues

vCheck fail - I am still seeing these disconnect/ssl (?) issues with the latest dev version. I can not figure out what the issue is, it seems adding/removing plugins sometimes helps but not always. Multiple environments so I don't think it's a one off thing. The latest build release (6.20) has no issues so as far as I can tell, something has been introduced that is causing the issues.

Can someone else grab a fresh copy and do some testing? I'd like to figure out where the issue is but need some more testers.

smasterson commented 10 years ago

Bump...

Is anyone else able to get the latest dev version to run consistently in their environment? I've tried four different environments now and all will bomb out with SSL/Disconnect errors. The errors appear in different plugins, at different times, even within the same environment. It's a very odd situation to say the least...

Sneddo commented 10 years ago

I've been running the latest dev copy for the last couple of days now, and not been able to replicate this fault :\ No matter what plugins I throw at it, I haven't had any SSL/Disconnect errors- including running with every plugin enabled.

Win7 x64 Powershell 2.0 PowerCLI 5.5.0.6316 vCenter 5.1 U1b

Scassillo commented 10 years ago

HI, i wanted to inform you that I also have this problem that occurs in a random. First I tested the latest version of vcheck Sphere on the following environment: W7 32 bit Powershell 4 PowerCLI 5.5 R2 patch1.

Now I've changed environment: Windows Server 2012 R2 Powershell 4 PowerCLI 5.5 R2 patch1 8 GB RAM

In both environments the error occurs.

SnakeNET64 commented 9 years ago

Hi, I have the same issue when I don't have enough free memory. I have a very large environment, and when no free RAM is available, the script hang and all i can do is disconnecting from vcenter Disconnect-VIServer -Force:$true -Confirm:$false Then, I kill some other programs, close my powershell script and relaunch vCheck. I don't have another solution :

kernelphr34k commented 9 years ago

Hello All,

I have used previous version of this script in the past and love it. I think the last version I used was 5, maybe older... Lots of great improvements to the current.. Thanks all.

My Env details: ~20 ESXi server ~600 VM's

Workstation Specs: Running vCheck: 6.23-alpha-1 PowerCLI 5.5 Realse 2 Patch 1 Powershell v3 Win7 x64 i7 w/ 24gb memory Win7 x64 i5 w/ 4gb mem...

I've only got this script working once with emailing me the results etc. All other times I get the error we are talking about and the script dies and never emails me. The only time the script worked for me It took 57min; Not sure if thats normal or not.

I have run many scripts via PowerCLI in the past and have never seen this error message like everyone else is seeing.. I did find another website that suggested some solutions, I have not tried myself but its worth trying... I will prolly test out the suggested solutions soon. Link here: http://d-fens.ch/2013/12/20/nobrainer-ssl-connection-error-when-using-powershell/

Anything else I can do to provide more info, or testing please let me know. Here's some screenshots.... The first shows the error after going through many plugins, the second shows the error while its in the middle of parsing thru VM's... Maybe yall can get more info from the screenshots... The last screenshot shows an error sending the email... wtf...

vcheck-error4 vcheck-error5 vcheck-error6

swerveshot commented 9 years ago

Hey @kernelphr34k,

To make troubleshooting easier for the plugins that return an error I suggest you run them seperately using the -Job parameter. Check out the documentation on the [vCheck wiki](../wiki/Job XML Specification).

Make sure to always include the following plugins: 00 Connection Plugin for vCenter.ps1 99 VeryLastPlugin Used to Disconnect.ps1

The error with the 'Sending Email' plugin looks like a problem with the number of parameters used to run the plugin. This could be a problem with your GlobalVariables file.

kernelphr34k commented 9 years ago

Thank you @rkleijwegt I will look into it.

This link works btw: https://github.com/alanrenouf/vCheck-vSphere/wiki/Job-XML-Specification

Sneddo commented 9 years ago

I suspect this disconnection issue is related to memory usage- the few times I've been able to replicate it have been when Powershell is using a lot of memory.

As for the email failure, try the suggestion from @rkleijwegt- it may be related to the other issues, or give more information than a fairly generic exception.

swerveshot commented 9 years ago

Yeah, I'm still learning how to use those fancy GitHub markdown codes. Fixed the link now. :smile:

kernelphr34k commented 9 years ago

@Sneddo What is considered high memory usage? If one of my machines has 24gb of memory it should not matter how much is used unless there is some Powershell limitations? You would think I would see a System.OutOfMemoryException error or something like it if it was a memory usage issue. Seeing a connection issue is weird imho....

I have been testing using @rkleijwegt suggestion but seeing another error about the given path's format is not supported as well as sending email issues.... Hope the screenshot helps...

vcheck-error7

Here is my job XML file:

<vCheck> 
    <globalVariables>GlobalVariables.ps1</globalVariables> 
        <plugins path="C:\scripts\vCheck-vSphere-master\Plugins"> 
        <plugin>\00 Initialize\00 Connection Plugin for vCenter.ps1</plugin>
        <plugin>\00 Initialize\01 General Information.ps1</plugin>
        <plugin>\80 Finish\999 VeryLastPlugin Used to Disconnect.ps1</plugin>
    </plugins> 
</vCheck>```

Don't get it.... should not be this difficult... :(

@rkleijwegt hah, its ok.... I have never needed a github account till now, so learning myself. Thanks!
smasterson commented 9 years ago

It appears as though it doesn't like the $Filename variable and then everything bombs out from there (cascading errors). Are you using the OutputPath parameter when running the script? If not, the script attempts to create an htm file in $Env:TEMP - any issues creating a file there manually?

[workaround] Have you tried sending as HTML (not as an attachment)? In GlobalVariables.ps1: $SendAttachment = $false

kernelphr34k commented 9 years ago

Hello @smasterson Not sure if I should have started a new thread or what.. I apologize if I should...

I have never used the OutputPath parameter. It should be sending out an html file as an attachment. No issues with $Env:TEMP can read/write there no issues.. there's other temp files there at the minute.

I tried to use the outputpath parameter and it seems to have worked (there is an html file in my output path), but still failed to send an email with the following settings:

vcheck-error8

I tried again without using the outputpath parameter and $SendAttachment = $false and got the following error:

vcheck-error9

Sneddo commented 9 years ago

@kernelphr34k I was seeing those disconnects when Powershell was using 1.5-2GB of memory. I agree that it seems odd- and quite possible that it was coincidence- but the only "unusual" thing I could see.

Really weird error you are getting with the temp directory. Can you try just putting in the following before line 737: Write-Host $FileName Curious to see why it is not in the correct format...

Timeout errors on the SMTP send is odd as well, I suppose it could be related to the file not saving correctly, but grasping as straws. I assume you can relay email from the machine you are running vCheck from?

kernelphr34k commented 9 years ago

Hi @Sneddo Unusual indeed..

So I put the Write-Host $FileName above line 737..... The file shows up in temp, I can open it, but still errors sending email. I have two SMTP servers I can use. I control one and have setup permissiosn to use.. The other is my works server and I have used this at one time and was able to get an email from it with a report. I have been switching between using both servers but mostly using my works since I had success with it once...

Screenshot shows the error and the html file it created... yet it does not attach and send.. odd.

vcheck-error10

$DisplaytoScreen = $false $SendEmail = $true $EmailReportEvenIfEmpty = $false $SendAttachment = $true

Thanks again!!

Sneddo commented 9 years ago

huh, that's even weirder! Maybe it's having issues with specifying the vCenter by IP, rather than hostname? Grasping at straws though...

Can you see the logs for either SMTP server? There might be a clue as to why it is failing from the server side, unfortunately the client error is a bit useless :(

djzang commented 9 years ago

I've been hitting this as well:

The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

The weird part is it seems to work fine on the first run but I will get this issue on subsequent runs of the script until the host I'm running powershell on reboots. I've tried it on two different systems with the same results

System 1: Windows 7 Ent SP1 64-bit VMware vSphere PowerCLI 5.8 Release 1 build 2057893 PSVersion 2.0 vCenter 5.0 Build 1300600

System 2: Windows 2008 R2 VMware vSphere PowerCLI 5.1 Release 2 Patch 1 build 1926866 PSVersion 2.0 vCenter 5.0 Build 1300600

I do have 44 VMKernel Warnings disabled but that doesn't seem to stop the errors from popping up. It's very weird how inconsistent the errors are.

kernelphr34k commented 9 years ago

@Sneddo hahaha odd I know!!!!! So these issues I'm having may be due to my network env. I'm in R/D, but they have us on a production network with tons of firewall rules afaik..

I copied the entire vCheck folder over to my vCenter server and it seems to be working fine using one SMTP server, the other not getting the emails but looking at logs. This should be my last msg on this thread as my issue has gone way beyond the initial issue problem.. Thanks for the help and working with me!!!

SnakeNET64 commented 9 years ago

Hello, For the problem about "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. "

I have a crappy workaround (vCheck.ps1 - Starting line 642)

$vCheckPlugins | Foreach { $TableFormat = $null try{ $PluginInfo = Get-PluginID $.Fullname $p++ Write-CustomOut ($lang.pluginStart -f $PluginInfo["Title"], $PluginInfo["Author"], $PluginInfo["Version"], $p, $vCheckPlugins.count) $pluginStatus = ($lang.pluginStatus -f $p, $vCheckPlugins.count, $.Name) Write-Progress -ID 1 -Activity $lang.pluginActivity -Status $pluginStatus -PercentComplete (100*$p/($vCheckPlugins.count)) $TTR = [math]::round((Measure-Command {$Details = . $_.FullName}).TotalSeconds, 2)

  Write-CustomOut ($lang.pluginEnd -f $PluginInfo["Title"], $PluginInfo["Author"], $PluginInfo["Version"], $p, $vCheckPlugins.count)

  $PluginResult += New-Object PSObject -Property @{"Title" = $PluginInfo["Title"];
                                                   "Author" = $PluginInfo["Author"];
                                                   "Version" = $PluginInfo["Version"];
                                                   "Details" = $Details;
                                                   "Display" = $Display;
                                                   "TableFormat" = $TableFormat;
                                                   "Header" = $Header;
                                                   "Comments" = $Comments;
                                                   "TimeToRun" = $TTR; }

*} catch { Write-Output "Plugin "+($PluginInfo["Title"])+" has failed... Ignoring result" -ForegroundColor red -BackgroundColor black Disconnect-VIServer -Force:$true -Confirm:$false Connect-VIServer $VIServer }**

}

PaulWalkerUK commented 9 years ago

I quite like this idea of catching errors from plugins in vCheck.ps1 and handling them cleanly.

From my own personal perspective, I use vCheck for a couple of different systems, but they're not VM-related, so I replace the whole plugins folder with my own. So for me, I would prefer it if vCheck.ps1 was kept generic and didn't have calls to things like disconnect/connect.

Maybe there could be some kind of special "recovery" plugin that would contain code like this (or whatever recovery code other variations might need) that would only be called if an error was detected here?

Sneddo commented 9 years ago

Yeah, agreed. Needs a bit of a tweak (as @PaulWalkerUK mentioned, we should keep vCheck as generic as possible), but I like the idea of at least gracefully catching the error until we can work out why we are getting this issue.

Sneddo commented 9 years ago

So, I stumbled across this today: http://blogs.technet.com/b/heyscriptingguy/archive/2013/07/30/learn-how-to-configure-powershell-memory.aspx

I wonder if bumping up the MaxMemoryPerShellMB setting will help with this issue... anyone that consistently gets this error able to test?

jones-g commented 9 years ago

@Sneddo

I have just configured the max memory to 2048MB and will see if this solves the problem when tonights reports run. Im not getting it consistently but almost every night on report fails.

djzang commented 9 years ago

@Sneddo I set max memory to 4096 using: winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="4096"}' Not sure if I'm setting it correctly?? However, I'm still getting the error: image

kernelphr34k commented 9 years ago

@Sneddo - Good find! My MaxMemoryPerShellMB was set to 150mb, set it to 2048mb. I tried to run vCheck and got the error again. So I set the MaxProcessesPerShell to 25, exited the powershell command prompt and reopened. Tried vCheck again and no errors. I'm doing a job with a lot less modulus, but there was a time I was seeing that error on my job.

jones-g commented 9 years ago

@Sneddo @kernelphr34k tried setting MaxMemoryPerShellMB to 2048 and the report still failed this evening. I will try setting the MaxProcessesPerShell to 35 (mine is 25 by default).

djzang commented 9 years ago

After changing MaxProcessesPerShell from 15 to 30 it looks like I'm not seeing the error anymore. I'll keep testing.

jones-g commented 9 years ago

@Sneddo @kernelphr34k @djzang I set it to MaxProcessesPerShell to 35 but that did not help. If I run the report now there is now problem but running it at 1.30 during the night fails. I have tried moving the scheduled task to execute earlier in the night and if that does not help I will try setting it for early in the morning instead.

SnakeNET64 commented 9 years ago

Maybe your scheduled task did not use the same account as you. I use another account to perform the query than everyday work. I change the max process value to 25 (and memory...) and the problem disappeared

Le 12 mars 2015 à 08:32, Jonas G notifications@github.com a écrit :

@Sneddo @kernelphr34k @djzang I set it to MaxProcessesPerShell to 35 but that did not help. If I run the report now there is now problem but running it at 1.30 during the night fails. I have tried moving the scheduled task to execute earlier in the night and if that does not help I will try setting it for early in the morning instead.

— Reply to this email directly or view it on GitHub.

jones-g commented 9 years ago

@Leleu256 I asked the Task Scheduler in Windows to run it for me exactly the same as if it triggered at a specified time so it is running as the same user.

Sneddo commented 9 years ago

hmmmm interesting results... what version of Powershell is everyone running? Let's see how this goes over the next few days.

DaveBF commented 9 years ago

During my testing of v6.22, I attempted to run this plugin and it ran for 3-4 hours before I just finally killed the script. I was thinking at the time it just took a long time to run and I didn't want the script running for that long.

Running Powershell v4 and PowerCLI 5.8 Release 1

JoJack82 commented 9 years ago

I set my MaxMemoryPerShellMB 4096mb and MaxProcessesPerShell to 35 but I still get this error. Anyone else have have a solution?

Get-Datacenter : 7/23/2015 3:26:26 PM Get-Datacenter The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

danimalrowe commented 9 years ago

Modified script to resolve issue #217 at my site for script "79 Find VMs in Uncontrolled Snapshot Mode.ps1". The change was made to search each Datastore for all VM directories that have files named like delta.vmdk and -*-flat.vmdk. In doing the file search this way there is only 2 calls per Datastore and only getting the information needed not other files that are not needed. This eliminates calling Get-Datacenter and retrieving all files for each VM. I have some questions about the following line of code in the current script

if ($file.Name -like '*delta.vmdk*' -or $file -like '-*-flat.vmdk') {

Should the 'or' portion of the statement be $file or $file.Name? I am assuming it should be $file.Name. And is the file to start with '-' (a hyphen)?

At my site the current version of "79 Find VMs in Uncontrolled Snapshot Mode.ps1" it ran for 2:40:03 and with this version it runs in 0:16:20 and I was able to run it with all the full vCheck.ps1 run. At my sight we do not have any uncontrolled snapshots showing up with either version so I would appreciate it if someone could verify that it works the same as the original version.

Our site consists of the following Number of Datastores: 67 Active VMs: 603

I had tried setting the Memory size and Timeout length and they seemed to help sometimes and not others. With this script I have set everything back to original settings and it runs fine.

See Pull Request #417

Thank you

stacycarter commented 8 years ago

I've been hitting the "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel" error intermittently whenever I try and run PowerCLI scripts from a 2008 R2 server running PowerCLI R1 (was able to recreate errors from two different 2008 R2 scripting servers, and against two different vCenter 5.5 U2/3 servers). Tried a couple of tweaks that were talked about in this thread, but that did not resolve the issue. I recently moved, my scripts to a 2012 R2 server running PowerCLI 6.0 R3, and it appears that this resolved the issue.

Sneddo commented 8 years ago

Interesting...are the server specs the same, and same version of WMF?

stacycarter commented 8 years ago

Both servers have plenty of resources (ie CPU, Memory). Same version of WMF.
Question for those that ran into this same TLS/SSL intermittent errors issue - does this fix work for you as well (ie running script from 2012 R2 server with PowerCLI 6.0 R3)?

meoso commented 8 years ago

I'm having this problem without a true resolve triggered by 84 VMDK consistency.ps1, 109 Orphaned VMDK File.ps1, and 79 Find VMs in Uncontrolled Snapshot Mode.ps1 (uncertain of consistency)

I've tried the following: $SendAttachment = $false even though i prefer the inverse

winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="3072"}'
winrm set winrm/config/winrs '@{MaxProcessesPerShell="100"}'

However, if i create a job XML, with only those plugins, they run without fail.

Therfore I am thinking that possibly releasing/clearing of variables is needed in between plugins.