powershell/windows-shell provisioner hangs on lengthy scripts

[gildas]

I have a powershell script that takes roughly 12 minutes to complete successfully. When I execute it through a powershell provisioner, like (in packer.json):

 {
   "type": "powershell",  "script": "./scripts/install-thing.ps1",
   "elevated_user": "vagrant", 
   "elevated_password": "vagrant"
 },  

Packer will at some point hang, not show any Write-Output/Write-Host the script emits, and not return anymore (I have to hit ^C to stop the build).

Does this ring a bell to anybody?

• The builder does not matter, I can reproduce this in Virtualbox, VMWare, and Parallels.
• I also have the same issue with the standard shell provisioner (over openssh).
• I did configure winrm in the VM properly (I think), see the gist: config-winrm.cmd

[mefellows]

No I've not had/seen that issue before but it sounds nasty. Are you able to share the details of the script you're running and/or the output of a build with the packer debug logs enabled?

[gildas]

Let me collect some fresh logs.

In the meantime, here is the gist of the script: install-icserver.ps1 If you execute this script inside the VM directly, it takes about 12 minutes. And you see all Write-Output messages. If you execute it from an Invoke-Command on a Windows machine remotely, it also works well. In the packer provisioner, you will see the first Write-Output in the packer output and log, then all Write-Output after the Start-Process will never get displayed. packer sits there doing nothing forever. I have to ^C it eventually. The packer log does not show anything either.

I cannot give you access to the actual MSI as this is not opensource/freeware/etc, sorry.

[mefellows]

Thanks for this @gildas, I suspect it has something to do with the Start-Process command running inside of a scheduled task (this looks related: http://stackoverflow.com/questions/23140588/powershell-start-process-not-running-as-scheduled-task). Need to ask some Powershell experts on this one.

[gildas]

Ah, ah, so maybe there would be a better way to run that msiexec.

Sorry for the logs, I was off last week and away from computers. And post-vacation wars since then. I hope the sky will clear up in a couple of days (funny given we had a typhoon over Tokyo last night). I hope to get something to you soon.

[mefellows]

No worries.

Potentially you could run it like cmd /c msiexec /qb /norestart /i ..., that should block until its finished as per my understanding of your script. It would still be nice in any case to understand if a Start-Process can be used in a scheduled task, and how that might be done.

[gildas]

Just for the sake of it, I will try without the Start-Process. I might look into chocolatey PoSH code as well as I noticed some inline choco install with the powershell provisioner take sometime well over 6 minutes, and it does not hang.

[mefellows]

I was literally just trawling through https://github.com/chocolatey/chocolatey/blob/master/src/helpers/functions/Start-ChocolateyProcessAsAdmin.ps1 and https://github.com/chocolatey/chocolatey/blob/master/src/helpers/functions/Install-ChocolateyPackage.ps1 to see how they do it. Creating a Choco package is a nice option as it makes provisioning scripts nice and clean.

Interestingly, the Start-ChocolateyProcessAsAdmin command does create a process, just not with Start-Process.

[gildas]

Interesting, it seems that back in time, Rob was using a Start-Process but migrated to that Start-ChocolateyProcessAsAdmin posh. In https://github.com/chocolatey/chocolatey/blob/master/src/helpers/functions/Install-ChocolateyInstallPackage.ps1,

Lines 70 & 71: Start-ChocolateyProcessAsAdmin "$msiArgs" 'msiexec' -validExitCodes $validExitCodes

Start-Process -FilePath msiexec -ArgumentList $msiArgs -Wait

He commented out the Start-Process.

[gildas]

Ok, I have updated the script using System.Diagnostics.ProcessStartInfo like in chocolatey. To get some idea, I wrote some crude elapsed time counter. Here is updated gist: install-icserver.ps1

It turns out the script does not output any text after 5 minutes (it should show elapsed time until the process is done) While the install actually succeeds, packer stays frozen waiting forever.

The packer log simply shows the outputs from the script and then nuttin' after 5 minutes...

Any thoughts?

[mefellows]

Sorry for delay on this @gildas, I'm hoping to free up some time next week to look at this in detail.

[mefellows]

OK, so I've managed to create a scenario installing 7zip that should replicate the bits and pieces we are talking about here. I used the Virtualbox OVF builder to save on OS install etc.

Unfortunately this installs just fine. I will attempt something that actually requires root privileges shortly to see if we can get a repro.

One thing I noticed is that on line 42 you refer to InstallSource but on line 14 there is a reference to the variable InstallPath which I assume is the intended variable. Not sure if that's the source of the issues but it certainly prevents the application from installing (script still returns though!).

I have tested this via the latest HEAD and the 1.0.0 release.

[gildas]

Hey no worries. My current workaround is to start the process, end the script, wait for 15 minutes (the average time it takes to run it), and check for completion for about 5 minutes (Weird, the check also is limited in time, now that I think about it.).

[gildas]

To answer your question, the gist I gave you was not a complete paste, seems I forgot a few things.

InstallPath is where the install goes and is passed to variables on the msiexec line (like: TRACING_LOGS="$InstallPath\Logs" While InstallSource is actually calculated after the SourceDriveLetter (something like: $Installsource = $(SourceDriveLetter} + ':\Install\ServerComponents').

Sorry about the confusion.

[gildas]

I think I will write a new check test and let it run more than 5 minutes. Keep you posted.

[mefellows]

No worries at all, thought i'd mention just in case (you never know!). I'd love to get to the bottom of it, it seems like this could be something likely to affect others.

[gildas]

Thanks for the help anyway! I want to test if I can reproduce this with a simpler script (like the one I use to check if the install worked by forcing the loop to fail over long enough time.).

BTW, I will need these tools with Hyper-V and Qemu. If you guys are not working on it already, I will start doing some work in a fork and submit a PR. Not top priority yet though.

[mefellows]

No worries at all.

There is a Qemu builder at packer.io if that's what you're after, and also there looks to have been an attempt at https://github.com/MSOpenTech/packer-hyperv/ for the Hyper-V stuff although I'm not sure how active/mature they are. That being said, we would certainly welcome any additions to the Packer Community!

[mefellows]

Closing this as I cannot reproduce. At the very least, we've proven that elevated lengthy tasks can be run (as per gist above). Thanks.

[Skarlso]

EDIT: SOLVED!! :+1:

I just updated to Packer 0.8.1. and everything works wonderfully. Awesomeness!

Damn, I ran into this stuff!!! Installing 7zip with Bat, or anything else, made it hang. And I ran into it just right now and had to re-write my whole packing because provisioning on a longer script did not work. Or even running this BAT file did not work:

cmd /c choco install curl -y binroot -version 0.04 -y apache.ant -y tortoisesvn -y -params 'ADDLOCAL=ALL' git -params '/GitOnlyOnPath' -y tortoisegit -y dotpeek -y

setx /m PATH "%PATH%;c:\Program Files (x86)\Git\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\TortoiseGit\bin"

This will hang on binroot. Binroot will install nicely but packer will not move on.

Not sure if it is maybe related to this bug with Packer => https://github.com/mitchellh/packer/issues/2100

However, this script isn't considered to be lengthy. I mean, choco install curl runs in, like, under a minute.