search

jetbrains-infra / packer-builder-vsphere

Packer plugin for remote builds on VMware vSphere
Mozilla Public License 2.0
541 stars 175 forks source link

WinRM not working #92

Closed rgl closed 6 years ago

rgl commented 6 years ago

For some reason the packer build gets stuck in the Waiting for WinRM to become available phase:

packer build -only=windows-2016-amd64-vsphere -on-error=abort windows-2016.json
windows-2016-amd64-vsphere output will be in this color.

==> windows-2016-amd64-vsphere: Creating VM...
==> windows-2016-amd64-vsphere: Adding CDRoms...
==> windows-2016-amd64-vsphere: Creating floppy disk...
    windows-2016-amd64-vsphere: Copying files flatly from floppy_files
    windows-2016-amd64-vsphere: Copying file: autounattend.xml
    windows-2016-amd64-vsphere: Copying file: vmtools.ps1
    windows-2016-amd64-vsphere: Copying file: winrm.ps1
    windows-2016-amd64-vsphere: Done copying files from floppy_files
    windows-2016-amd64-vsphere: Collecting paths from floppy_dirs
    windows-2016-amd64-vsphere: Resulting paths from floppy_dirs : []
    windows-2016-amd64-vsphere: Done copying paths from floppy_dirs
==> windows-2016-amd64-vsphere: Uploading created floppy image
==> windows-2016-amd64-vsphere: Adding generated Floppy...
==> windows-2016-amd64-vsphere: Adding configuration parameters...
==> windows-2016-amd64-vsphere: Power on VM...
==> windows-2016-amd64-vsphere: Waiting 10s for boot...
==> windows-2016-amd64-vsphere: Typing boot command...
==> windows-2016-amd64-vsphere: Waiting for IP...
==> windows-2016-amd64-vsphere: IP address: 10.0.0.177
==> windows-2016-amd64-vsphere: Waiting for WinRM to become available...

Doing a packet capture, its stuck sending these http requests:

POST /wsman HTTP/1.1
Host: 10.0.0.177:5985
User-Agent: Go-http-client/1.1
Content-Length: 1340
Authorization: Basic dmFncmFudDp2YWdyYW50
Content-Type: application/soap+xml;charset=UTF-8
Accept-Encoding: gzip

<?xml version="1.0" encoding="utf-8" ?>
<env:Envelope 
    xmlns:env="http://www.w3.org/2003/05/soap-envelope" 
    xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" 
    xmlns:rsp="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" 
    xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" 
    xmlns:p="http://schemas.microsoft.com/wbem/wsman/1/wsman.xsd">
    <env:Header>
        <a:To>http://10.0.0.177:5985/wsman</a:To>
        <a:ReplyTo>
            <a:Address mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</a:Address>
        </a:ReplyTo>
        <w:MaxEnvelopeSize mustUnderstand="true">153600</w:MaxEnvelopeSize>
        <w:OperationTimeout>PT2H</w:OperationTimeout>
        <a:MessageID>uuid:c87669d5-b913-4493-693e-624532b8572c</a:MessageID>
        <w:Locale mustUnderstand="false" xml:lang="en-US"/>
        <p:DataLocale mustUnderstand="false" xml:lang="en-US"/>
        <a:Action mustUnderstand="true">http://schemas.xmlsoap.org/ws/2004/09/transfer/Create</a:Action>
        <w:ResourceURI mustUnderstand="true">http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</w:ResourceURI>
        <w:OptionSet>
            <w:Option Name="WINRS_NOPROFILE">FALSE</w:Option>
            <w:Option Name="WINRS_CODEPAGE">65001</w:Option>
        </w:OptionSet>
    </env:Header>
    <env:Body>
        <rsp:Shell>
            <rsp:InputStreams>stdin</rsp:InputStreams>
            <rsp:OutputStreams>stdout stderr</rsp:OutputStreams>
        </rsp:Shell>
    </env:Body>
</env:Envelope>

HTTP/1.1 200 
Content-Type: application/soap+xml;charset=UTF-8
Server: Microsoft-HTTPAPI/2.0
Date: Thu, 26 Apr 2018 18:50:58 GMT
Content-Length: 1624

<s:Envelope xml:lang="en-US" 
    xmlns:s="http://www.w3.org/2003/05/soap-envelope" 
    xmlns:a="http://schemas.xmlsoap.org/ws/2004/08/addressing" 
    xmlns:x="http://schemas.xmlsoap.org/ws/2004/09/transfer" 
    xmlns:w="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd" 
    xmlns:rsp="http://schemas.microsoft.com/wbem/wsman/1/windows/shell" 
    xmlns:p="http://schemas.microsoft.com/wbem/wsman/1/wsman.xsd">
    <s:Header>
        <a:Action>http://schemas.xmlsoap.org/ws/2004/09/transfer/CreateResponse</a:Action>
        <a:MessageID>uuid:F905D3A4-2DD5-4017-81DB-97493243057B</a:MessageID>
        <a:To>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</a:To>
        <a:RelatesTo>uuid:c87669d5-b913-4493-693e-624532b8572c</a:RelatesTo>
    </s:Header>
    <s:Body>
        <x:ResourceCreated>
            <a:Address>http://10.0.0.177:5985/wsman</a:Address>
            <a:ReferenceParameters>
                <w:ResourceURI>http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</w:ResourceURI>
                <w:SelectorSet>
                    <w:Selector Name="ShellId">2F5C53A0-93D6-4D2A-9074-89B73238E3D9</w:Selector>
                </w:SelectorSet>
            </a:ReferenceParameters>
        </x:ResourceCreated>
        <rsp:Shell 
            xmlns:rsp="http://schemas.microsoft.com/wbem/wsman/1/windows/shell">
            <rsp:ShellId>2F5C53A0-93D6-4D2A-9074-89B73238E3D9</rsp:ShellId>
            <rsp:ResourceUri>http://schemas.microsoft.com/wbem/wsman/1/windows/shell/cmd</rsp:ResourceUri>
            <rsp:Owner>vagrant</rsp:Owner>
            <rsp:ClientIP>10.0.0.176</rsp:ClientIP>
            <rsp:IdleTimeOut>PT7200.000S</rsp:IdleTimeOut>
            <rsp:InputStreams>stdin</rsp:InputStreams>
            <rsp:OutputStreams>stdout stderr</rsp:OutputStreams>
            <rsp:ShellRunTime>P0DT0H0M0S</rsp:ShellRunTime>
            <rsp:ShellInactivity>P0DT0H0M0S</rsp:ShellInactivity>
        </rsp:Shell>
    </s:Body>
</s:Envelope>

That message is the first winrm message to start a shell, but packer never seems to send a command to that shell... any idea why?

Note that manually using the winrm command (from https://github.com/masterzen/winrm-cli) it works fine:

winrm -username vagrant -password vagrant -hostname 10.0.0.177 -port 5985 whoami
vagrant-4e4futn\vagrant

I'm using the latest packer-builder-vsphere-iso.exe binary from https://teamcity.jetbrains.com/viewLog.html?buildId=1404924&buildTypeId=PackerVSphere_Build&tab=artifacts (build 29 from git revision https://github.com/jetbrains-infra/packer-builder-vsphere/commit/1d61fa430b67693b97f10f25ad7edca0d57028c5)

rgl commented 6 years ago

Forgot to include relevant packer log lines.

Here, while it cannot connect to WinRM it sucessfully retries and outputs these types of messages

2018/04/26 19:47:56 packer-builder-vsphere-iso.exe: 2018/04/26 19:47:56 [INFO] Attempting WinRM connection...
2018/04/26 19:47:56 packer-builder-vsphere-iso.exe: 2018/04/26 19:47:56 [DEBUG] connecting to remote shell using WinRM
2018/04/26 19:47:56 packer-builder-vsphere-iso.exe: 2018/04/26 19:47:56 [ERROR] connection error: http response error: 401 - invalid content type
2018/04/26 19:47:56 packer-builder-vsphere-iso.exe: 2018/04/26 19:47:56 [ERROR] WinRM connection err: http response error: 401 - invalid content type

And finally, when WinRM is finally configured in the VM, it starts to output these errors:

2018/04/26 19:48:01 packer-builder-vsphere-iso.exe: 2018/04/26 19:48:01 [INFO] Attempting WinRM connection...
2018/04/26 19:48:01 packer-builder-vsphere-iso.exe: 2018/04/26 19:48:01 [DEBUG] connecting to remote shell using WinRM
2018/04/26 19:48:02 packer-builder-vsphere-iso.exe: 2018/04/26 19:48:02 [ERROR] connection error: Malformed XML file
2018/04/26 19:48:02 packer-builder-vsphere-iso.exe: 2018/04/26 19:48:02 [ERROR] WinRM connection err: Malformed XML file

I'm not sure why its failing, because the XML from the packet capture seems correct.

rgl commented 6 years ago

Isn't https://github.com/jetbrains-infra/packer-builder-vsphere/blob/808a7ce57d77ecc4e988d11049e52efe17b5e4d0/iso/builder.go#L71 missing to set WinRMConfig? Or is WinRM still not supported in the latest multistep support added to this plugin?

rgl commented 6 years ago

BTW, with packer-builder-vsphere v2.0-beta4 it works fine. This issue is just about the master version.

VladRassokhin commented 6 years ago

with packer-builder-vsphere v2.0-beta4 it works fine

That's strange because nothing has changed around 'StepConnect' except updated Packer dependency. Will investigate further.

missing to set WinRMConfig?

WinRMConfig is optional and could be omitted. winrm_username and winrm_password would be used in that case.

VladRassokhin commented 6 years ago

Some search into packer issues and looking into code shows that error message is a bit incorrect, what's most probably happening is that Windows machine responds with 401 Unauthorized and some plain text. I'll submit pull request into winrm connector library with proper error message.

Please check that there's basic authentication enabled on machine and username and password specified in packer config.

stacycarter commented 6 years ago

I may have the same issue. Also building Windows 2016. Winrm was working fine for me with v2.0-beta4. My winrm config: in addition to specifying the username and password, I included the "winrm_use_ntlm": "true" line and set winrm "allowunencrypted" to true (did not set auth to basic and this is what worked with the beta consistently). As soon as I switched to the master version, it stopped working.

rgl commented 6 years ago

@VladRassokhin winrm is correctly configured (supports basic authentication and non-encrypted connections) and works fine on the vm as you can see by the manual use of winrm-cli (the same winrm code that packer uses) that I've mentioned in this issue:

Note that manually using the winrm command (from https://github.com/masterzen/winrm-cli) it works fine:

winrm -username vagrant -password vagrant -hostname 10.0.0.177 -port 5985 whoami
vagrant-4e4futn\vagrant
rgl commented 6 years ago

In case you need to test, my packer template and supporting scripts for creating a base Windows 2016 image are at https://github.com/rgl/windows-2016-vagrant/tree/vsphere (in the vsphere branch).

mkuzmin commented 6 years ago

I've confirmed the issue. ~So far it looks like a regression in Packer 1.2 we started using recently~.

mkuzmin commented 6 years ago

I believe I've fixed that. winrm library has switched XML parser in https://github.com/masterzen/winrm/pull/73. The new parser rejects XML documents without <?xml version="1.0" ?> header.

My fork disables the validation, but we still need to open PR there.

mkuzmin commented 6 years ago

No need for PRs. Just need to import a fresh revision of ChrisTrenkamp/goxpath