Powershell provisioning not working intermittently

[andrewsav-bt]

Issue described here: https://groups.google.com/forum/#!topic/packer-tool/i-zfeluWRY4

Occasionally Powershell provisioner silently fails to run the script. No error is generated and no script is run. This is what log looks like when this issue happens:

[00:38:53][Step 1/2] ==> windows-2012-base: Provisioning with Powershell...
[00:38:53][Step 1/2] ==> windows-2012-base: Provisioning with shell script: ./scripts/debloat-windows.ps1
[00:38:53][Step 1/2] ==> windows-2012-base: Provisioning with shell script: ./scripts/install-choco.ps1
[00:38:54][Step 1/2] ==> windows-2012-base: Provisioning with shell script: ./scripts/install-7z.ps1
[00:38:54][Step 1/2] ==> windows-2012-base: Provisioning with shell script: ./scripts/install-vmtools.ps1
[00:38:54][Step 1/2] ==> windows-2012-base: Provisioning with shell script: ./scripts/install-bginfo.ps1
[00:38:55][Step 1/2] ==> windows-2012-base: Provisioning with shell script: ./scripts/install-antivirus.ps1
[00:38:55][Step 1/2] ==> windows-2012-base: Restarting Machine
[00:38:55][Step 1/2] ==> windows-2012-base: Waiting for machine to restart...

When the issues does NOT happen the same part of the log looks like this:

https://gist.github.com/andrewsav-datacom/c59e876ad2bd9d8dab18749e1b900dd5

This is what the relevant part of the packer config looks like:

    {
      "type": "windows-restart"
    },
    {
      "type": "powershell",
      "valid_exit_codes": [ 0, 3010 ],
      "scripts": [
        "./scripts/debloat-windows.ps1",
        "./scripts/install-choco.ps1",
        "./scripts/install-7z.ps1",
        "./scripts/install-vmtools.ps1", 
        "./scripts/install-bginfo.ps1",
    "./scripts/install-antivirus.ps1"
      ]
    },
    {
      "type": "windows-restart"
    },

This is what packer log looks like when the issue is occurring:

https://gist.github.com/andrewsav-datacom/711e98a410f6194270f559fdcc6555d6

This issue is very hard to track, it does not happen often, but when it does it goes undetected because error is not generated. Also in the example above 6 scripts fail in a row, this is incredibly rare, usually it's just one or none.

[DanHam]

@andrewsav-datacom Could you let us know what version of Packer you are using? Please run packer version and paste the output. From the logs it looks as though you are not using the latest version.

Also, if you could please provide gists of the full logs and the template and scripts you are using, that would help a lot in trying to reproduce and understand the root cause of the failure.

[AndrewSav]

Yes, those particular logs were taken with 1.0.4 I have upgraded since then. Was there any changes in powershell provisioner between then and now? Do you think that full logs and the template and scripts will still be useful, or would you like to close the issues for now and let me file a new one when it happens again?

[DanHam]

@AndrewSav

Was there any changes in powershell provisioner between then and now?

Yes. There were a number of changes that have been made since then. However, the particular change that I think may help your situation is #5300 - this makes sure that the PowerShell provisioner is ready to go again after restarts. If you read through #5064 (one of the issues fixed by #5300) you'll see a lot of similarities to your situation.

I think this issue is a duplicate of the issues that were reported there. See the comments here and here that appear to describe the exact same issue that you are having.

Long and short, I think this will be fixed by upgrading to the latest version of Packer. However, @SwampDragons would be better placed to make that call as she tracked down the root cause of the issues reported in #5064 and coded up the fix in #5300... @SwampDragons - Neat work btw!

[SwampDragons]

Yeah, this was my suspicion too. Please let me know if upgrading to the latest version of packer solves your intermittent issues.

[AndrewSav]

@SwampDragons I will but it will take a few weeks. The packer build process (it's multi-steps with a chain of templates built one on top of the other) I'm running currently takes about 18 hours to run (with all the windows updates). I'm running it weekly. I will monitor the results in the next few weeks and I'll get back if I still have problems. Feel free to close the issue in the meanwhile, I'll be happy to open a new one if the problem persists.

[SwampDragons]

That's fair; if the issue reoccurs, please comment here again and I'll re-open it. :)