Powershell provisioner failing, not showing results and does not proceed.

[trodemaster]

I'm seeing some unexpected behavior with the winrm communicator and powershell provisioner. One of the first things I'm porting over is the VMware tools install process of my vmware-iso Windows Server 2012r2 packer template. At the point when vmware tools setup runs the packer console session just stops showing any output. On investigation of the vm I see log files created by the powershell provisioner and the script does in fact complete running as expected. As a test I connected to the vm via powershell enter-pssession and that connection continued to work while vmware tools was being installed. Debug logging in packer shows nothing and I even let it sit overnight once.

what the packer console shows.. vmware-iso: Installed VMware cert... vmware-iso: Installing VMWare Tools... vmware-iso: Waiting for VMware tools to install

Then nothing happens at this point and I verified that winrm sessions are not dropping and the system is still reachable.

If you look at the .out log file from the powershell provisioner run of the tools-win-new.ps1 script it's clearly finishing. https://gist.github.com/trodemaster/531b678131c2afbb2a2b

Anybody have suggestions or should I just get a bug report started?

Git repo for the packer project https://github.com/trodemaster/packer-WinSrv2012r2/tree/packer-winrm-tools-disconnect

WinSrv2012r2.json template file https://github.com/trodemaster/packer-WinSrv2012r2/blob/packer-winrm-tools-disconnect/WinSrv2012r2.json

I'm using a file provisioner to upload the VMware tools installer. Then this powershell script to install tools. https://github.com/trodemaster/packer-WinSrv2012r2/blob/packer-winrm-tools-disconnect/scripts/tools-win-new.ps1

Packer build debug log https://gist.github.com/trodemaster/5dacdd1e2072b65dccaf

[pecigonzalo]

Can you confirm your machine is not rebooting after the install? I suspect it migth not be taking all the arguments properly.

[trodemaster]

The VM does not reboot after installing VMware tools. Anytime I setup the tools start-process to wait the provisioner hangs. The process does return non zero exit status most of the time so that may be part of the issue. My plan was to handle the reboot with the new windows-restart provisioner running after tools install.

[pecigonzalo]

Yeah that should work, im using this to install the tools, let me know if you get a different outcome. https://gist.github.com/pecigonzalo/3e9f6ae2baf00cd2e93b

[trodemaster]

I have a workaround. Really looking for a fix to the packer behavior.

[pecigonzalo]

Well, we dont know if its a packer behavior, it might be something on your side. I think you are losing network connectivity to the guest and thats why you stop seeing output, your timeout is 2h. This could happen because you are installing the network driver maybe, could you try without that?

[trodemaster]

If I connect a remote powershell session to the vm before it starts the vmware tools installation it remains connected for the whole process.

On Tue, Jul 28, 2015 at 1:49 PM, Gonzalo Peci notifications@github.com wrote:

Well, we dont know if its a packer behavior, it might be something on your side. I think you are losing network connectivity to the guest and thats why you stop seeing output, your timeout is 2h. This could happen because you are installing the network driver maybe, could you try without that?

— Reply to this email directly or view it on GitHub https://github.com/mitchellh/packer/issues/2473#issuecomment-125750381.

[pecigonzalo]

Did you try my gist for installing the tools? most cases that i had like that it was the installer trying to show a gui at the other end.

[trodemaster]

I have verified that my tools install script work as expected using a remote powershell session and when used in my other templates that are ssh based.

On Wed, Jul 29, 2015 at 1:43 PM, Gonzalo Peci notifications@github.com wrote:

Did you try my gist for installing the tools? most cases that i had like that it was the installer trying to show a gui at the other end.

— Reply to this email directly or view it on GitHub https://github.com/mitchellh/packer/issues/2473#issuecomment-126090166.

[pecigonzalo]

Blake, unfortunately i cant replicate your issue, i tried with your vmware tool install script and provisioning configuration but it works just fine for me and continues provisioning just fine.

[trodemaster]

I just ran it again to confirm what I see. The installation succeeds and the installer exists with the expected 3010 status of pending reboot. You can see the full output of the powershell provisioner in the .out file. That data is not showing up in the packer terminal stream. After this provisioner finishes the packer build hangs. Here is a screenshot. http://blake-public.s3.amazonaws.com/Screen%20Shot%202015-07-30%20at%2010.58.20%20AM.png

[cbednarski]

@trodemaster I think if you leave this running long enough packer will timeout the connection to the machine and mark this as a failure.

However, I'm inclined to think this is weird behavior in your script. Can you add another script afterward to say "hello" or try rebooting when the install is complete? That might tell us whether packer is not receiving data vs. the script being stuck.

[trodemaster]

Yes let me spend some time on it next week. I'm about to head out of town for a couple days. At one point I had a windows-reboot after the powershell provisioner and left it running overnight. I'm still suspecting the error codes passed from the vmware tools installer are causing issue.

On Jul 30, 2015, at 5:15 PM, Chris Bednarski notifications@github.com wrote:

@trodemaster https://github.com/trodemaster I think if you leave this running long enough packer will timeout the connection to the machine and mark this as a failure.

However, I'm inclined to think this is weird behavior in your script. Can you add another script afterward to say "hello" or try rebooting when the install is complete? That might tell us whether packer is not receiving data vs. the script being stuck.

— Reply to this email directly or view it on GitHub https://github.com/mitchellh/packer/issues/2473#issuecomment-126527311.

[cbednarski]

@trodemaster If it helps there are some similar tool scripts here: https://github.com/joefitzgerald/packer-windows. I'm not super knowledgable about Windows, personally, so I'd use Joe's stuff if I did this.

[nickgold90]

I have had a look at this and I think it is related to using a VMXnet 3 driver prior to the packer build. When VMware tools is installed it does not completely install if the VMXnet3 driver is reinstalled. If you exclude the VMxnet3 driver then it seems to work.

[trodemaster]

I also believe it has something to do with the network or winrm disconnecting in an odd way. With my build the vmxnet3 driver is actually loaded at the winpe servicing stage before tools attempts to install. Currently I just kick off the install and have the script exit. Adding a 5 min wait before the next powershell provisioner allows it to pickup and do what's needed.

[cbednarski]

I have had a look at this and I think it is related to using a VMXnet 3 driver prior to the packer build.

Adding a 5 min wait before the next powershell provisioner allows it to pickup and do what's needed.

Interesting. I wonder if we should add a forced reconnect somewhere on packer's side, or if there is a better way to do a clean shutdown inside the VM before updating the driver (I may have misunderstood the failure case).

[trodemaster]

I have been doing a lot of winrm builds lately. This is the most reliable situation I can reproduce the winrm communicator going zombie on me. I have also seen situations when winrm will connect and start to process a new powershell provisioner and then just timeout. Right now I have lots of pause_before 1m with all of my powershell provisioners to eliminate that. Short of a packet capture I'm not sure what other info will help narrow this down.

[cbednarski]

This issue may be resolved pending an update and merge of https://github.com/mitchellh/packer/pull/3541.

[aaronk1]

+1 The full packer build works perfectly with VMware Tools installed from Workstation 12.1 Pro, using e1000e network interface and lsisas1068 disk controller. As soon as I change to vmxnet3 and pvscsi (using driver from floppy from ESXi 6's VMware Tools during WinPE stage of Windows Setup), Packer build hangs at VMware Tools install using winrm provisioner (using Workstation 12.1's VMware Tools).

I'll look at the workarounds posted. Thanks everyone for reporting this and providing some possible fixes.

[aaronk1]

@trodemaster What exactly did you do to work around the issue? I've tried the pause_before 1m in my PowerShell provisioner and I tried Windows Restart as a provisioner and neither work for me to continue the VMware Tools install (still hangs). Worst case, I'll build the VM with e1000e NIC and just run a script after VMware Tools installs to remove the NIC and readd the vmxnet3 one. When the VM is uploaded to vSphere and then deployed, the NIC is easy to change anyways.

[trodemaster]

So yes I think it's something with drivers and winrm getting into a funky state. My workaround is to start the tools install with powershell start-process but don't wait for it to exit. Then pause before the next provisioner for 5 minutes. Ugly but it works..

start-process -FilePath 'C:/Windows/Temp/setup64.exe' -ArgumentList '/S /v "/qn /l*v ""C:\windows\temp\vmwtoolsinstall.log"" ADDLOCAL=ALL REMOVE=Hgfs REBOOT=R"'

Hopefully I'll have some time to test if the current version of packer is still having the exact same issue and confirm my suspicion in a week or so. I do use vmxnet3 and pvscsi drivers in our builds as you do.

[nickgold90]

I change the NIC after the build in the VMX file. Alternatively I tried to run the VMware tools install without installing the vmxnet3 driver.

[aaronk1]

@cbednarski any idea when a release with #3541 will be out?

[mwhooker]

Going to close this issue as there's some notion that this is solved in a later version of packer. If this is still a problem, please submit a new issue with steps to reproduce. thanks!