XML syntax error on line 1: illegal character code U+001F

gou1 commented 9 years ago

Hey there,

I'm already using Terraform to manage AWS resources, and am now trying to manage some vSphere VMs with it. I found your provider here https://github.com/hashicorp/terraform/issues/102#issuecomment-135021149

I've set up the following:

Ubuntu 14.04 vagrant vm
Terraform v0.6.3
This provider (tried both 0.3.1 and pre-release, dl+unzip+mv in /usr/local/bin+chmod 0755)

I'm using the following TF configuration:

provider "vsphere" {
    user = "xxxxxx"
    password = "xxxxxx"
    vcenter_server = "xxxxxx"
}

resource "vsphere_virtual_machine" "default" {
    name = "xxxxxx"
    datacenter = "xxxxxx"
    cluster = "xxxxxx"
    vcpu = 1
    memory = 1024
    disk {
        datastore = "xxxxxx"
        template = "templates/UbuntuServ14"
    }
    gateway = "xxxxxx"
    network_interface {
        label = "eth0"
        ip_address = "xxxxxx"
        subnet_mask = "xxxxxx"
    }
}

But when i terraform apply i get:

Error applying plan:

1 error(s) occurred:

* vsphere_virtual_machine.default: error: XML syntax error on line 1: illegal character code U+001F

Any idea what I got wrong?

Thanks!

chrislovecnm commented 9 years ago

Did you complete any more debugging? I am wondering if this is a problem with the go library that is being used, and not the provider.

Thanks

Chris

gou1 commented 9 years ago

I'm not sure what kind of debug info would help, let me know if you need something precise.

For now: I installed govc (from https://github.com/vmware/govmomi/releases), configured the env variables GOVC_URL/USERNAME/PASSWORD and ran govc about:

$ govc about
Name:         VMware vCenter Server
Vendor:       VMware, Inc.
Version:      5.5.0
Build:        2183111
OS type:      win32-x64
API type:     VirtualCenter
API version:  5.5
Product ID:   vpx
UUID:         xxxxxxxxxxxxxxx

chrislovecnm commented 9 years ago

It would be good to get where the U+001F character is coming from and which line of code is throwing the error. It is in an XML payload, so either the vSphere go api library is not happy or the provider is not happy. Let me look at it a bit and get back to you. I need to get a new vCenter lab up anyways.

chrislovecnm commented 9 years ago

I am guessing that the vSphere go api is throwing an error here:

https://github.com/vmware/govmomi/blob/a31c97a9acc012fb2f6faca441e78f0c3d1e67df/vim25/xml/xml.go#L1069

My hex math is REALLY rusty but I think the test is failing at line https://github.com/vmware/govmomi/blob/a31c97a9acc012fb2f6faca441e78f0c3d1e67df/vim25/xml/xml.go#L1086

If this is true the API is functioning as designed. The question would be why is vSphere giving you that XML character.... Will look into it more.

@tkak you have any advice on debugging this??

chrislovecnm commented 9 years ago

@gou1 can you follow the instructions here https://www.terraform.io/docs/internals/debugging.html and get the logs into a gist? Clear out the logs completely, and just get me logs to the error. The api has a command line tool and we can probably recreate it using that tool, if we know what terraform is doing. I am 'guessing' we are getting a weird UTF character from vSphere. But I need more info.

Merci mon ami.

p.s. don't go to French as you will loose this Canadian very very very quickly.

gou1 commented 9 years ago

Et voilà:

2015/09/17 17:14:40 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* vsphere_virtual_machine.default: error: XML syntax error on line 1: illegal character code U+001F
2015/09/17 17:14:40 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* vsphere_virtual_machine.default: error: XML syntax error on line 1: illegal character code U+001F
2015/09/17 17:14:40 [ERROR] root: eval: *terraform.EvalOpFilter, err: 1 error(s) occurred:

* vsphere_virtual_machine.default: error: XML syntax error on line 1: illegal character code U+001F
2015/09/17 17:14:40 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* vsphere_virtual_machine.default: error: XML syntax error on line 1: illegal character code U+001F

The log itself is 2000+ lines, I quickly checked and couldn't find anything useful - but then again go/terraform internals/hex are not my core skills :laughing:. I'll setup a gist with the full logs if you need.

Also in the logs there's:

2015/09/17 17:14:40 terraform-provider-vsphere: 2015/09/17 17:14:40 [INFO] VMWare vSphere Client configured for URL: https://xxx:xxx@xxx/sdk

If y make an empty POST request to that URL, i get a SOAP error which rules out (i guess) networking issues:

<?xml version="1.0" encoding="UTF-8" ?>
<soapenv:Envelope>
<soapenv:Body>
 <soapenv:Fault>
<faultcode>ClientFaultCode</faultcode>
<faultstring>Error returned by expat parser: no element found while parsing HTTP request before method was determined at line 1, column 0</faultstring>
 </soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>

I would have liked to capture http packets, but it's https so tcpdump won't help here.

tkak commented 9 years ago

@chrislovecnm Thank you for your help.

@gou1 I am guessing that the govmomi issue. Can you try to run govc command in debug mode? And please check XML files.

$ govc ls -debug
$ ls ~/.govmomi/debug/xxxx

chrislovecnm commented 9 years ago

@gou1 some more details for you. Here is the readme for govc https://github.com/vmware/govmomi/tree/master/govc - the 'ls' command will debug if govc is configured correctly and is able to communicate to vSphere. The readme provides details on how to configure govc to make the ls call into vSphere.

The https post call you made will fail. Being an API you have to send xml payloads to specific urls, as it is a web service api.

@tkak and I are thinking that the problem is at the govmomi or vSphere level. If govmomi is not configured correctly to talk to vSphere it may have a problem like this, or if govmomi has a bug then you may want to file a bug with them. I am working on learning more about govmomi and this terraform providers, so I am interested in working through this. The govc ls -debug should give us more information. The odd thing is that govmoni is catching a odd UTF character.

chrislovecnm commented 9 years ago

@gou1 any update??

gou1 commented 9 years ago

I first ran:

$ govc ls -debug
Error: ServerFaultCode: Cannot complete login due to an incorrect user name or password.

I checked the xml files and found a POST request body with an empty tag (even though GOVC_PASSWORD was set as it should ; and also the username was fine in the request body). I ran the command with the credentials as -u option rather than env variables and got:

$ govc ls -debug -u xxx:xxx@example.com
/xxxxxxx_datacenter1457/vm
/xxxxxxx_datacenter1457/network
/xxxxxxx_datacenter1457/host
/xxxxxxx_datacenter1457/datastore

Looking at the debug XML files, the responses seem okay.

chrislovecnm commented 9 years ago

@tkak you think looking into his logs are a good next step? I am thinking that we need to determine which call contains the XML error.

tkak commented 9 years ago

Hmmm..., I need to more info about the XML error. But, I think we should report the issue and the debug XM files to govmomi for resolving the issue smoothly.

@gou1 Could you create a new issue in govmomi?

lekoder commented 8 years ago

I seem to be having same problem:

koder@koder:~/Workspace/Code/terraform$ terraform apply
vsphere_virtual_machine.default: Creating...
  disk.#:                          "" => "1"
  disk.0.iops:                     "" => "500"
  disk.0.size:                     "" => "1"
  domain:                          "" => "vsphere.local"
  memory:                          "" => "1024"
  name:                            "" => "terraform_default"
  network_interface.#:             "" => "1"
  network_interface.0.ip_address:  "" => "<computed>"
  network_interface.0.label:       "" => "production"
  network_interface.0.subnet_mask: "" => "<computed>"
  time_zone:                       "" => "Etc/UTC"
  vcpu:                            "" => "2"
Error applying plan:

1 error(s) occurred:

* vsphere_virtual_machine.default: XML syntax error on line 1: illegal character code U+001F

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

koder@koder:~/Workspace/Code/terraform$ terraform --version
Terraform v0.6.8

govc on the other hand seems to work without problem:

koder@koder:~/Workspace/Code/terraform$ govc ls --debug
/pcc-xxx-xxx-xxx-xxx/vm
/pcc-xxx-xxx-xxx-xxx/network
/pcc-xxx-xxx-xxx-xxx/host
/pcc-xxx-xxx-xxx-xxx/datastore

I got following debug data:

koder@koder:~/Workspace/Code/terraform$ ls ~/.govmomi/debug/2015-12-07T10-11-26.482332752/
1-0001.req.headers  1-0001.res.xml      2-0001.req.xml      2-0002.req.headers  2-0002.res.xml      2-0003.res.headers  2-0004.req.xml      2-0005.req.headers  2-0005.res.xml      2-0006.res.headers
1-0001.req.xml      1-client.log        2-0001.res.headers  2-0002.req.xml      2-0003.req.headers  2-0003.res.xml      2-0004.res.headers  2-0005.req.xml      2-0006.req.headers  2-0006.res.xml
1-0001.res.headers  2-0001.req.headers  2-0001.res.xml      2-0002.res.headers  2-0003.req.xml      2-0004.req.headers  2-0004.res.xml      2-0005.res.headers  2-0006.req.xml      2-client.log

...and there doesn't seem to be anything wrong with them:

koder@koder:~/.govmomi/debug/2015-12-07T10-11-26.482332752$ xmllint --noout *.xml
koder@koder:~/.govmomi/debug/2015-12-07T10-11-26.482332752$

I fiddled around with govc and everything seems to work just fine. Is there any additional data i could provide to help fix this problem?

Mykolaichenko commented 8 years ago

I have the same problem:

mykolaichenko:vsphere mukolaich$ terraform apply
vsphere_virtual_machine.haproxy: Creating...
  cluster:                                "" => "Cluster2"
  datacenter:                             "" => "xxxx"
  disk.#:                                 "" => "1"
  disk.0.datastore:                       "" => "xxx"
  disk.0.template:                        "" => "ubuntu-14.04.04-0.4"
  domain:                                 "" => "vsphere.local"
  gateway:                                "" => "10.10.18.1"
  memory:                                 "" => "4096"
  name:                                   "" => "haproxy.xx.prod.un.private"
  network_interface.#:                    "" => "1"
  network_interface.0.ip_address:         "" => "<computed>"
  network_interface.0.ipv4_address:       "" => "10.10.18.101"
  network_interface.0.ipv4_prefix_length: "" => "24"
  network_interface.0.ipv6_address:       "" => "<computed>"
  network_interface.0.ipv6_prefix_length: "" => "<computed>"
  network_interface.0.label:              "" => "VLAN18"
  network_interface.0.subnet_mask:        "" => "<computed>"
  resource_pool:                          "" => "m2"
  time_zone:                              "" => "Etc/UTC"
  vcpu:                                   "" => "2"
Error applying plan:

1 error(s) occurred:

* vsphere_virtual_machine.haproxy: XML syntax error on line 1: illegal character code U+001F

Terraform does not automatically rollback in the face of errors. Instead, your Terraform state file has been partially updated with any resources that successfully completed. Please address the error above and apply again to incrementally change your infrastructure.

Mykolaichenko commented 8 years ago

Here is end of TRACE:

2016/03/29 12:24:31 [DEBUG] apply: vsphere_virtual_machine.test: executing Apply
vsphere_virtual_machine.test: Creating...
  disk.#:                                 "" => "1"
  disk.0.template:                        "" => "ubuntu-14.04.04-0.4"
  disk.0.type:                            "" => "eager_zeroed"
  domain:                                 "" => "vsphere.local"
  memory:                                 "" => "4096"
  name:                                   "" => "terraform_test"
  network_interface.#:                    "" => "1"
2016/03/29 12:24:31 [DEBUG] terraform-provider-vsphere: 2016/03/29 12:24:31 [DEBUG] network_interface init: [{ VM_Network  0  0 }]
2016/03/29 12:24:31 [DEBUG] terraform-provider-vsphere: 2016/03/29 12:24:31 [DEBUG] disk init: [{0 0 eager_zeroed}]
  network_interface.0.ip_address:         "" => "<computed>"
  network_interface.0.ipv4_address:       "" => "<computed>"
  network_interface.0.ipv4_prefix_length: "" => "<computed>"
  network_interface.0.ipv6_address:       "" => "<computed>"
  network_interface.0.ipv6_prefix_length: "" => "<computed>"
  network_interface.0.label:              "" => "VM_Network"
  network_interface.0.subnet_mask:        "" => "<computed>"
  time_zone:                              "" => "Etc/UTC"
  vcpu:                                   "" => "2"
2016/03/29 12:24:31 [DEBUG] root: eval: *terraform.EvalWriteState
2016/03/29 12:24:31 [DEBUG] root: eval: *terraform.EvalApplyProvisioners
2016/03/29 12:24:31 [DEBUG] root: eval: *terraform.EvalIf
2016/03/29 12:24:31 [DEBUG] root: eval: *terraform.EvalWriteDiff
2016/03/29 12:24:31 [DEBUG] root: eval: *terraform.EvalIf
2016/03/29 12:24:31 [DEBUG] root: eval: *terraform.EvalWriteState
2016/03/29 12:24:31 [DEBUG] root: eval: *terraform.EvalApplyPost
2016/03/29 12:24:31 [ERROR] root: eval: *terraform.EvalApplyPost, err: 1 error(s) occurred:

* vsphere_virtual_machine.test: XML syntax error on line 1: illegal character code U+001F
2016/03/29 12:24:31 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* vsphere_virtual_machine.test: XML syntax error on line 1: illegal character code U+001F
2016/03/29 12:24:31 [ERROR] root: eval: *terraform.EvalOpFilter, err: 1 error(s) occurred:

* vsphere_virtual_machine.test: XML syntax error on line 1: illegal character code U+001F
2016/03/29 12:24:31 [ERROR] root: eval: *terraform.EvalSequence, err: 1 error(s) occurred:

* vsphere_virtual_machine.test: XML syntax error on line 1: illegal character code U+001F
2016/03/29 12:24:31 [TRACE] [walkApply] Exiting eval tree: vsphere_virtual_machine.test
2016/03/29 12:24:31 [DEBUG] vertex provider.vsphere (close), got dep: vsphere_virtual_machine.test
2016/03/29 12:24:31 [DEBUG] waiting for all plugin processes to complete...
Error applying plan:

1 error(s) occurred:

* vsphere_virtual_machine.test: XML syntax error on line 1: illegal character code U+001F

I think its throwed here: https://github.com/vmware/govmomi/blob/8f2d79fd740b69c4f64d10e8ac0b8c6b30247c0d/vim25/xml/xml.go#L36

Maybe I can replace or remove this character from XML? Thanks a lot.

rakutentech / terraform-provider-vsphere

XML syntax error on line 1: illegal character code U+001F #31