chocolatey / boxstarter

Repeatable, reboot resilient windows environment installations made easy using Chocolatey packages
https://boxstarter.org/
Apache License 2.0
1.29k stars 163 forks source link

Timeout ignored when explicitly specified for Chocolatey package install #214

Open JorritSalverda opened 7 years ago

JorritSalverda commented 7 years ago

When installing a large Chocolatey package remotely on Windows Server 2016 the execution timeout passed to Chocolatey seems to be overridden by the shorter BoxStarter timeout.

I run the following command with a 3 hour timeout

cinst visualstudio2015community --version=2015.03.02 --execution-timeout=10800

However the installation fails after an hour with the Boxstarter: Task has exceeded its total timeout. error:

18:15:54.663     googlecompute: [PACKER-580F9DC6]+ Boxstarter starting Calling Chocolatey to install visualstudio2015community. This may take several minutes to complete...
18:15:54.663     googlecompute: Installing the following packages:
18:15:54.664     googlecompute: visualstudio2015community
18:15:54.664     googlecompute: By installing you accept licenses for the packages.
18:15:56.242     googlecompute: VisualStudio2015Community v2015.03.02
18:15:59.814     googlecompute: Visual Studio will be installed from Web
18:15:59.815     googlecompute: Install-ChocolateyPackage VisualStudio2015Community exe /Quiet /NoRestart /Log C:\Users\packeradmin\AppData\Local\Temp\vs.log http://download.microsoft.com/download/0/B/C/0BC321A4-013F-479C-84E6-4A2F90B11269/vs_community.exe -validExitCodes 0 3010 2147781575
18:16:00.892     googlecompute: Downloading VisualStudio2015Community
18:16:00.893     googlecompute: from 'http://download.microsoft.com/download/0/B/C/0BC321A4-013F-479C-84E6-4A2F90B11269/vs_community.exe'
18:16:00.893     googlecompute: Download of vs_community.exe (2.9 MB) completed.
18:16:04.112     googlecompute: WARNING: Ignoring checksums due to feature checksumFiles turned off or option --ignore-checksums set.
18:16:04.114     googlecompute: [PACKER-580F9DC6]Boxstarter: Expanded splat to -validExitCodes @(0,3010,2147781575) -packageName "VisualStudio2015Community" -fileType "exe" -silentArgs "/Quiet /NoRestart /Log C:\Users\packeradmin\AppData\Local\Temp\vs.log" -file "C:\Users\packeradmin\AppData\Local\Temp\VisualStudio2015Community\2015.03.02\vs_community.exe"
18:16:10.189     googlecompute: Installing VisualStudio2015Community...
19:16:07.206     googlecompute: [PACKER-580F9DC6]Boxstarter: Task has exceeded its total timeout. Killing task...
19:16:08.627     googlecompute: TASK:
19:16:08.628     googlecompute: Import-Module C:\ProgramData\chocolatey\helpers\chocolateyInstaller.psm1 -Global -DisableNameChecking
19:16:08.628     googlecompute: Install-ChocolateyInstallPackage -validExitCodes @(0,3010,2147781575) -packageName "VisualStudio2015Community"
19:16:08.629     googlecompute: -fileType "exe" -silentArgs "/Quiet /NoRestart /Log C:\Users\packeradmin\AppData\Local\Temp\vs.log" -file
19:16:08.629     googlecompute: "C:\Users\packeradmin\AppData\Local\Temp\VisualStudio2015Community\2015.03.02\vs_community.exe"
19:16:08.630     googlecompute: Is likely in a hung state.
19:16:08.631     googlecompute: At C:\Users\packeradmin\AppData\Roaming\Boxstarter\Boxstarter.Common\Invoke-FromTask.ps1:186 char:17
19:16:08.631     googlecompute: + ...             throw "TASK:`r`n$command`r`n`r`nIs likely in a hung state ...
19:16:08.632     googlecompute: +                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
19:16:08.633     googlecompute: + CategoryInfo          : OperationStopped: (TASK:
19:16:08.633     googlecompute: Import-M...n a hung state.:String) [], RuntimeException
19:16:08.634     googlecompute: + FullyQualifiedErrorId : TASK:
19:16:08.635     googlecompute: Import-Module C:\ProgramData\chocolatey\helpers\chocolateyInstaller.psm1 -Global -DisableNameChecking
19:16:08.637     googlecompute: Install-ChocolateyInstallPackage -validExitCodes @(0,3010,2147781575) -packageName "VisualStudio2015Community" -fi
19:16:08.638     googlecompute: leType "exe" -silentArgs "/Quiet /NoRestart /Log C:\Users\packeradmin\AppData\Local\Temp\vs.log" -file "C:\Users\p
19:16:08.639     googlecompute: ackeradmin\AppData\Local\Temp\VisualStudio2015Community\2015.03.02\vs_community.exe"
19:16:08.641     googlecompute: Is likely in a hung state.
19:16:08.641     googlecompute: The install of visualstudio2015community was NOT successful.
19:16:08.642     googlecompute: Error while running 'C:\ProgramData\chocolatey\lib\VisualStudio2015Community\Tools\ChocolateyInstall.ps1'.
19:16:08.642     googlecompute: See log for details.
19:16:11.002     googlecompute: Chocolatey installed 0/1 package(s). 1 package(s) failed.
19:16:11.003     googlecompute: Failures:
19:16:11.004     googlecompute: - visualstudio2015community

Is there a BoxStarter specific way to increase the timeout? And could BoxStarter respect the --execution-timeout parameter instead?

By the way, it could be that the installation really hangs. But I think my winrm settings are already tweaked to prevent its limitations being in the way:

Config
    MaxEnvelopeSizekb = 500
    MaxTimeoutms = 60000
    MaxBatchItems = 32000
    MaxProviderRequests = 4294967295
    Client
        NetworkDelayms = 5000
        URLPrefix = wsman
        AllowUnencrypted = false
        Auth
            Basic = true
            Digest = true
            Kerberos = true
            Negotiate = true
            Certificate = true
            CredSSP = false
        DefaultPorts
            HTTP = 5985
            HTTPS = 5986
        TrustedHosts
    Service
        RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
        MaxConcurrentOperations = 4294967295
        MaxConcurrentOperationsPerUser = 1500
        EnumerationTimeoutms = 240000
        MaxConnections = 300
        MaxPacketRetrievalTimeSeconds = 120
        AllowUnencrypted = false
        Auth
            Basic = true
            Kerberos = true
            Negotiate = true
            Certificate = false
            CredSSP = false
            CbtHardeningLevel = Relaxed
        DefaultPorts
            HTTP = 5985
            HTTPS = 5986
        IPv4Filter = *
        IPv6Filter = *
        EnableCompatibilityHttpListener = false
        EnableCompatibilityHttpsListener = false
        CertificateThumbprint
        AllowRemoteAccess = true
    Winrs
        AllowRemoteShellAccess = true
        IdleTimeout = 7200000
        MaxConcurrentUsers = 2147483647
        MaxShellRunTime = 2147483647
        MaxProcessesPerShell = 2147483647
        MaxMemoryPerShellMB = 2147483647
        MaxShellsPerUser = 2147483647
JorritSalverda commented 7 years ago

When running the same command via remote desktop it takes 41 minutes:

$ Get-Date

Wednesday, October 26, 2016 8:04:26 AM

$ cinst visualstudio2015community --version=2015.03.02 --execution-timeout=10800 --yes --accept-license

Chocolatey v0.10.3
Installing the following packages:
visualstudio2015community
By installing you accept licenses for the packages.

VisualStudio2015Community v2015.03.02 [Approved]
visualstudio2015community package files install completed. Performing other installation steps.
System.Collections.Hashtable
Visual Studio will be installed from Web
Install-ChocolateyPackage VisualStudio2015Community exe /Quiet /NoRestart /Log C:\Users\jsalverda\AppData\Local\Temp\cho
colatey\vs.log http://download.microsoft.com/download/0/B/C/0BC321A4-013F-479C-84E6-4A2F90B11269/vs_community.exe -valid
ExitCodes 0 3010 2147781575
WARNING: Url has SSL/TLS available, switching to HTTPS for download
Downloading VisualStudio2015Community
  from 'https://download.microsoft.com/download/0/B/C/0BC321A4-013F-479C-84E6-4A2F90B11269/vs_community.exe'
Progress: 100% - Completed download of C:\Users\jsalverda\AppData\Local\Temp\chocolatey\VisualStudio2015Community\2015.0
3.02\vs_community.exe (2.9 MB).
Download of vs_community.exe (2.9 MB) completed.
Hashes match.
Installing VisualStudio2015Community...
VisualStudio2015Community has been installed.
No ISO to dismount
Environment Vars (like PATH) have changed. Close/reopen your shell to
 see the changes (or in powershell/cmd.exe just type `refreshenv`).
 The install of visualstudio2015community was successful.
  Software installed to 'C:\Program Files\Microsoft SQL Server Compact Edition\'

Chocolatey installed 1/1 packages. 0 packages failed.
 See the log for details (C:\ProgramData\chocolatey\logs\chocolatey.log).

$ Get-Date

Wednesday, October 26, 2016 8:45:10 AM
JorritSalverda commented 7 years ago

Inspecting the vs.log it seems like it doesn't do anything after the first couple of seconds.

[0F28:0DE0][2016-10-26T08:35:12]i001: Burn v3.7.4029.0, Windows v10.0 (Build 14393: Service Pack 0), path: C:\Users\packeradmin\AppData\Local\Temp\VisualStudio2015Community\2015.03.02\vs_community.exe, cmdline: '/Quiet /NoRestart /Log C:\Users\packeradmin\AppData\Local\Temp\vs.log  -burn.unelevated BurnPipe.{853EB3DD-F94C-4E92-95FA-1B437B2CA8EF} {C85253E4-052D-4431-A028-40531C902BCD} 3588'
[0F28:0DE0][2016-10-26T08:35:12]i000: Initializing string variable 'SKUFriendlyName' to value 'VS Community'

...

[0F28:0DE0][2016-10-26T08:35:17]i000: MUX:  Scheduling update from http://go.microsoft.com/fwlink/?LinkID=786519&clcid=0x409.
[0F28:0DE0][2016-10-26T08:35:17]i000: MUX:  Executing Process: Exe:C:\Users\packeradmin\AppData\Local\Temp\20161026_083517_{4312D170-FE25-36BF-B5E6-0A87C44B7EF0}\vs_community.exe Arguments: /OriginalSource "C:\Users\packeradmin\AppData\Local\Temp\VisualStudio2015Community\2015.03.02\vs_community.exe" /UpdateLaunch /ProductKey "WXN74VRMXHJ8X3HM8F7WCPQB8" /SessionGuid "be43c22a-f8f8-4590-a97a-56f860103fd9" /quiet /norestart
mwrock commented 7 years ago

When boxstarter installs MSIs remotely, it runs them in a sheduled task and sets a timeout on the task of one hour. If the task does not complete in an hour, it kills the task. Sounds like you must be hitting this timeout. Unfortunately there is not currently a way to override that however, it sounds to me like there is something fundamentally wrong with the vs install if it is taking this long.

JorritSalverda commented 7 years ago

It seems to be related to server 2016. Haven't tested yet with visualstudio2015community, but the following older visual studio package works fine on server 2012 r2 but hangs on server 2016:

cinst VisualStudio2013ExpressWeb --version 12.0.21005.20150920 --ignore-checksums

Perhaps some incompatibility warning or the by default enabled windows defender kicking in. Via remote desktop this works fine, but unfortunately not over winrm. Any tips on how to find the root cause of a hang?

dragon788 commented 7 years ago

@JorritSalverda It is entirely possible that there are some pre-dependencies like vcredist and other things that don't exist for Server 2016 that the older version of Visual Studio think they require. Do you have any output from the chocolatey.log file output into the $env:ProgramData folder where Chocolatey gets installed? Not sure if that file gets created when Boxstarter uses the DLL or not.

pauby commented 5 years ago

Would be good to look at if we can change the scheduled task timeout mentioned here if there is no other way around this.